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Preface to the Second Edition 

The Office of Planning, Research and Evaluation (OPRE), a unit within the Administration for Children and Families (ACF), 
is responsible for advising the Assistant Secretary for Children and Families on increasing the effectiveness and efficiency 
of programs to improve the economic and social well-being of children and families. 

In collaboration with ACF program offices and others, OPRE is responsible for performance management for ACF, 
conducts research and policy analyses, and develops and oversees research and evaluation projects to assess program 
performance and inform policy and practice. The Office provides guidance, analysis, technical assistance, and oversight 
to ACF programs on: strategic planning; performance measurement; research and evaluation methods; statistical, policy, 
and program analysis; and synthesis and dissemination of research and demonstration findings. 

Toward these ends, OPRE is happy to present an updated edition of The Program Manager's Guide to Evaluation. The 
original guide has consistently been the most frequently accessed of OPRE's resources. The new edition has been 
updated, under the guidance of Kathleen Dwyer, to reflect currently accepted practices, up-to-date terminology, and 
issues to consider at this time. Tips, samples, and worksheets that were missing from the online version have been 
inserted back into the guide, as has a thoroughly updated appendix containing a comprehensive list of evaluation 
resources. Based on feedback within ACF, we have chosen to focus on a single guide that includes examples that would 
be relevant to all ACF program areas, rather than create separate handbooks for each program. 

As with the original edition of The Program Manager's Guide to Evaluation, this updated edition explains what program 
evaluation is, why evaluation is important, how to conduct an evaluation and understand the results, how to report 
evaluation findings, and how to use evaluation results to improve programs that benefit children and families. 

Naomi Goldstein 
Director 

Office of Planning, Research and Evaluation 
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IN THIS CHAPTER... 
Benefits of evaluation 
Guidelines for success 
Purpose of this manual 


Why Evaluate 
Your Program? 

Y ou should evaluate your program because an evaluation 
helps you accomplish the following: 

■ Find out what is and is not working in your program 

■ Show your funders and the community what your 
program does and how it benefits your participants 

■ Raise additional money for your program by providing 
evidence of its effectiveness 

■ Improve your staffs work with participants by 
identifying weaknesses as well as strengths 

■ Add to the existing knowledge in the human services 
field about what does and does not work in your type 
of program with your kinds of participants 

Despite these important benefits, program managers often are 
reluctant to evaluate their programs. Usually this reluctance is due 
to concerns stemming from a lack of understanding about the 
evaluation process. 

COMMON CONCERNS ABOUT EVALUATION 

Concern #1: Evaluation diverts resources away from the 
program and therefore harms participants. This is a common 
concern in most programs. However, because evaluation helps to 
determine what does and does not work in a program, it is actually 
beneficial to program participants. Without an evaluation, you are 
providing services with little or no evidence that they actually 
work! 
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MONITORING asks whether a 
program meets the operating 
standards set by the funder, State, 
or licensing agency. 

EVALUATION asks whether 
the program has been successful 
in achieving its objectives for 
program implementation and 
participant outcomes. 


Concern # '2, : Evaluation increases the burden for program 
staff. Often program staff are responsible for collecting evaluation 
information because they are most familiar with, and have the 
most contact with program participants. Despite this potential for 
increased burden, staff can benefit gready from evaluation because 
it provides information that can help them improve their work 
with participants, learn more about program and participant needs, 
and validate their successes. Also, the burden can be decreased 
somewhat by incorporating evaluation activities into ongoing 
program activities. 

Concern #3: Evaluation is too complicated. Program managers 
often reject the idea of conducting an evaluation because they 
don't know how to do it or whom to ask for help. Although the 
technical aspects of evaluation can be complex, the evaluation 
process itself simply systematizes what most program managers 
already do on an informal basis - figure out whether the program's 
objectives are being met, which aspects of the program work, and 
which ones are not effective. Understanding this general process 
will help you to be a full partner in the evaluation, even if you seek 
outside help with the technical aspects. If you need outside help, 
Chapter 4 provides some ideas about how and where to get it. 

Concern #■ 4 : Evaluation may produce negative results and 
lead to information that will make the program look bad or 
lose funding. An evaluation may reveal problems in 
accomplishing the work of the program as well as successes. It is 
important to understand that both types of information are 
significant. The discovery of problems should not be viewed as 
evidence of program failure, but rather as an opportunity to learn 
and improve the program. Information about both problems and 
successes not only helps your program, but also helps other 
programs learn and improve. When evaluation results are used for 
accountability purposes, it is all the important that the evaluation 
be carefully planned and executed. 

Concern #5; Evaluation is just another form of monitoring. 

Program managers and staff often view program evaluation as a 
way for funders to monitor programs to find out whether staff are 
doing what they are supposed to be doing. Program evaluation, 
however, is not the same as monitoring. Sometimes the 
information collected to monitor a program overlaps with 
information needed for an evaluation, but the two processes ask 
very different questions. 
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Concern #• 6 Evaluation requires setting performance 
standards, and this is too difficult. Many program managers 
believe that an evaluation requires setting performance standards, 
such as specifying the percentage of participants who will 
demonstrate changes or exhibit particular behaviors. Program staff 
worry that if these performance standards are not met, their 
project will be judged a failure. 

This concern is somewhat justified because often funders will 
require setting such standards. However, performance standards 
can only be set if there is extensive evaluation information on a 
particular program in a variety of settings. Without this 
information, performance standards are completely arbitrary and 
meaningless. The type of evaluation discussed in this manual is not 
designed to assess whether particular performance standards are 
attained because most programs do not have sufficient information 
to establish these standards in any meaningful way. Instead, it will 
assess whether there has been significant change in the knowledge, 
attitudes, and/or behaviors of a program's participant population 
in general and whether particular characteristics of the program or 
the participants are more or less likely to promote change. 

GUIDELINES FOR CONDUCTING A SUCCESSFUL 
EVALUATION 

You can maximize the benefits that evaluation offers by following 
a few basic guidelines in preparing for and conducting your 
evaluation. 

Invest heavily in planning. Invest both time and effort in 
deciding what you want to learn from your evaluation. This is the 
single most important step you will take in this process. Consider 
what you would like to discover about your program and its impact 
on participants, and use this information to guide your evaluation 
planning. 

Integrate the evaluation into ongoing activities of the 
program. Program managers often view evaluation as something 
that an outsider "does to" a program after it is over, or as an 
activity "tacked on" merely to please funders. Unfortunately, many 
programs are evaluated in this way. This approach greatly limits the 
benefits that program managers and staff can gain from an 
evaluation. Planning the evaluation should begin at the same time 
as planning the program so that you can use evaluation feedback to 
inform program operations. 
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Participate in the evaluation and show program staff that you 
think it is important. An evaluation needs the participation of the 
program manager to succeed. Even if an outside evaluator is hired 
to conduct the evaluation, program managers must be full partners 
in the evaluation process. An outside evaluator cannot do it alone. 
You must teach the evaluator about your program, your 
participants, and your objectives. Also, staff will value the 
evaluation if you, the program manager, value it yourself. Talk 
about it with staff individually and in meetings. If you hire an 
outside evaluator to conduct the evaluation, be sure that this 
individual attends staff meetings and gives presentations on the 
status of the evaluation. Your involvement will encourage a sense 
of ownership and responsibility for the evaluation among all 
program staff. 

Involve as many of the program staff as much as possible and 
as early as possible. Project staff have a considerable stake in the 
success of the evaluation, and involving them early on in the 
process will enhance the evaluation's effectiveness. Staff will have 
questions and issues that the evaluation can address, and are 
usually pleased when the evaluation validates their own hunches 
about what does and does not work in the program. Because of 
their experiences and expertise, program staff can ensure that the 
evaluation questions, design, and methodology are appropriate for 
the program's participants. Furthermore, early involvement of staff 
will promote their willingness to participate in data collection and 
other evaluation-related tasks. 

Be realistic about the burden on you and your staff. 

Evaluations are work. Even if your evaluation calls for an outside 
evaluator to do most of the data collection, it still takes time to 
arrange for the evaluator to have access to records, administer 
questionnaires, or conduct interviews. It is common for both 
agencies and evaluators to underestimate how much additional 
effort this involves. When program managers and staff brainstorm 
about all of the questions they want answered, they often produce 
a very long list. This process can result in an evaluation that is too 
complicated. Focus on the key questions that assess your 
program's general effectiveness. 

Be aware of the ethical and cultural issues in an evaluation. 

This guideline is very important. When you are evaluating a 
program that provides services or training, you must always 
consider your responsibilities to the participants and the 
community. You must ensure that the evaluation is relevant to and 
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respectful of the cultural backgrounds and individuality of 
participants. Evaluation instruments and methods of data 
collection must be culturally sensitive and appropriate for your 
participants. Participants must be informed that they are taking 
part in an evaluation and that they have the right to refuse to 
participate in this activity without jeopardizing their participation 
in the program. Finally, you must ensure that confidentiality of 
participant information will be maintained. 

ABOUT THIS MANUAL 

This manual is designed to help you follow these guidelines while 
planning and implementing a program evaluation. Each of the 
chapters addresses specific steps in the evaluation process and 
provides guidance on how to tailor an evaluation to your program's 
needs. 

The manual is not intended to turn you into a professional 
evaluator or to suggest that evaluation is a simple process that 
anyone can perform. Rather, it is meant to provide information to 
help you understand each step of the evaluation process so that 
you can participate fully in the evaluation- whether you hire an 
outside evaluator or decide to do one with assistance from in- 
house agency staff and resources. 
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What Is Program 
Evaluation? 


IN THIS CHAPTER... 
Types of program evaluation 
Steps in the process 
Cost considerations 


P rogram managers and staff frequently informally assess 
their program's effectiveness: Are participants benefiting 
from the program? Are there sufficient numbers of 
participants? Are the strategies for recruiting participants 
working? Are participants satisfied with the services or training? 
Do staff have the necessary skills to provide the services or 
training? These are all questions that program managers and staff 
ask and answer on a routine basis. 

Evaluation addresses these same questions, but uses a systematic 
method for collecting, analyzing, and using information to answer 
basic questions about a program - and to ensure that those answers 
are supported by evidence. This does not mean that conducting an 
evaluation requires no technical knowledge or experience - but it 
also does not mean that evaluation is beyond the understanding of 
program managers and staff. 

WHAT ARE THE BASIC QUESTIONS AN 
EVALUATION CAN ANSWER? 

There are many different types of program evaluations, many 
different terms to describe them, and many questions that they can 
answer. You may have heard the terms formative evaluation, 
summative evaluation, process evaluation, outcome evaluation, 
cost-effectiveness evaluation, and cost-benefit evaluation. 
Definitions of these terms and others and selected resources for 
more information on various types of program evaluations are 
provided in the appendix. 

You may have also heard the terms "qualitative" and "quantitative" 
used to describe an evaluation. However, these terms, which are 
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DEFINITION OF PROGRAM 
EVALUATION: Program 
evaluation is a systematic method 
for collecting, analyzing, and 
using information to answer basic 
questions about a program. 


defined in the glossary, refer to the types of information or data 
that are collected during the evaluation and not to the type of 
evaluation itself. For example, an outcome evaluation may involve 
collecting both quantitative and qualitative information about 
participant outcomes. 

This manual is designed to avoid the confusion that often results 
from the use of so many terms to describe an evaluation. Instead, 
all of the terms used here are directly related to answering 
evaluation questions derived from a program's objectives. 

There are two types of program objectives - program 
implementation objectives and participant outcome 
objectives. Program implementation objectives refer to what you 
plan to do in your program, how you plan to do it, and who you 
want to reach. They include the services or training you plan to 
implement, the characteristics of the participant population, the 
number of people you plan to reach, the staffing arrangements and 
staff training, and the strategies for recruiting participants. 
Evaluating program implementation objectives is often referred to 
as a process evaluation. However, because there are many types of 
process evaluations, this manual will use the term 
implementation evaluation. 

Participant outcome objectives describe what you expect to 
happen to your participants as a result of your program, with the 
term "participants" referring to agencies, communities, and 
organizations as well as individuals. Your expectations about how 
your program will change participants' knowledge, attitudes, 
behaviors, or awareness are your participant outcome objectives. 
Evaluating a program's success in attaining its expectations for 
participants is often called an outcome evaluation. 

An evaluation can be used to determine whether you have been 
successful in attaining both types of objectives, by answering the 
following questions: 

Y Has the program been successful in attaining the 
anticipated implementation objectives? (Are you 
implementing the services or training that you initially planned 
to implement? Are you reaching the intended target 
population? Are you reaching the intended number of 
participants? Are you developing the planned collaborative 
relationships?) 
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MAXIMIZING EVALUATION 
BENEFITS: Evaluations that are 
incorporated as an integral part of 
ongoing program operations 
provide optimum benefits to 
managers, staff, and participants. 


V Has the program been successful in attaining the 
anticipated participant outcome objectives? (Are 
participants exhibiting the expected changes in knowledge, 
attitudes, behaviors, or awareness? Can these changes be 
attributed to the program?) 

A comprehensive evaluation must answer both key questions. You 
may be successful in attaining your implementation objectives, but 
if you do not have information about participant outcomes, you 
will not know whether your program is worthwhile. Similarly, you 
may be successful in changing participants' knowledge, attitudes, or 
behaviors; but if you do not have information about your 
program's implementation, you will be unable to identify the parts 
of your program that are associated with these changes. 

These evaluation questions should be answered while a program is 
in operation, not after the program is over. This approach will 
allow you and your staff to identify problems and make necessary 
changes while the program is still operational. It will also ensure 
that program participants are available to provide information for 
the evaluation. 

Although not discussed in depth in this manual, two other 
categories of questions may be of interest: questions regarding the 
need for services (needs assessment) and questions regarding the 
program’s efficiency (efficiency assessment). A needs assessment is 
a study of the problem that a program is intended to address and 
the need for the program. An efficiency assessment is a study that 
measures program costs and compares them to either a monetary 
value of the program’s benefit (cost-benefit analysis) or a measure 
of the program’s effectiveness in achieving its outcome objectives 
(cost-effectiveness analysis). 

WHAT IS INVOLVED IN CONDUCTING AN 
EVALUATION? 

The term "systematic" in the definition of evaluation indicates that 
it requires a structured and consistent method of collecting and 
analyzing information about your program. You can ensure that 
your evaluation is conducted in a systematic manner by following a 
few basic steps. 

Step 1: Assemble an evaluation team. Planning and executing an 
evaluation should be a team effort. Even if you hire an outside 
evaluator or consultant to help, you and members of your staff 
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BASIC EVALUATION STEPS 

Step 1: Assemble an evaluation 
team 

Step 2: Prepare for the evaluation 

Step 3: Develop an evaluation 
plan. 

Step 4: Collect evaluation 
information. 

Step 5: Analyze your evaluation 
information. 

Step 6: Prepare the evaluation 

report. 


must be full partners in the evaluation effort. Chapter 3 discusses 
various evaluation team options. If you plan to hire an outside 
evaluator or an evaluation consultant, Chapter 4 provides 
information on hiring procedures and managing an evaluation that 
involves an outside professional. 

Step 2: Prepare for the evaluation. Before you begin, you will 
need to build a strong foundation. This planning phase includes 
deciding what to evaluate, building a program model, stating your 
objectives in measurable terms, and identifying the context for the 
evaluation. The more attention you give to planning the evaluation, 
the more effective it will be. Chapter 5 will help you prepare for 
your evaluation. 

Step 3: Develop an evaluation plan. An evaluation plan is a 
blueprint or a map for an evaluation. It details the design and the 
methods that will be used to conduct the evaluation and analyze 
the findings. You should not implement an evaluation until you 
have completed an evaluation plan. Information on what to 
include in a plan is provided in Chapter 6. 

Step 4: Collect evaluation information. Once you complete an 
evaluation plan, you are ready to begin collecting information. This 
task will require selecting or developing information collection 
procedures and instruments. This process is discussed in Chapter 
7. 

Step 5: Analyze your evaluation information. After evaluation 
information is collected, it must be organized in a way that allows 
you to analyze it. Information analysis should be conducted at 
various times during the course of the evaluation to allow you and 
your staff to obtain ongoing feedback about the program. This 
feedback will either validate what you are doing or identify areas 
where changes may be needed. Chapter 8 discusses the analysis 
process. 

Step 6: Prepare the evaluation report. The evaluation report 
should be a comprehensive document that describes the program 
and provides the results of the information analysis. The report 
should also include an interpretation of the results for 
understanding program effectiveness. Chapter 9 is designed to 
assist you in preparing an evaluation report. 


9 




THE PROGRAM MANAGER’S GUIDE TO EVALUATION 


A useful rule of thumb is to 
estimate that your evaluation will 
cost approximately 1 5 to 20 
percent of your total program 
budget. 


WHAT WILL AN EVALUATION COST? 

Program managers are often concerned about the cost of an 
evaluation. This is a valid concern. Evaluations do require money. 
Many program managers and staff believe that it is unethical to use 
program or agency financial resources for an evaluation, because 
available funds should be spent on serving participants. However, 
it is more accurate to view money spent on evaluation as an 
investment in your program and in your participants, rather than as 
a diversion of funds available for participants. Evaluation is 
essential if you want to know whether your program is benefiting 
participants. 

Unfortunately, it is not possible to specify in this manual exactly 
how much money you will need to conduct your evaluation. The 
amount of money needed depends on a variety of factors, 
including what aspects of your program you decide to evaluate, the 
size of the program (that is, the number of staff members, 
participants, components, and services), the number of outcomes 
that you want to assess, who conducts the evaluation, and your 
agency's available evaluation-related resources. Costs also vary in 
accord with economic differences in communities and geographic 
locations. 

Sometimes funders will establish a specific amount of grant money 
to be set aside for an evaluation. The amount usually ranges from 
15 to 20 percent of the total funds allocated for the program. If the 
amount of money to be set aside for an evaluation is not specified 
by a funding agency, you may want to talk to other program 
managers in your community who have conducted evaluations. 
They may be able to tell you how much their evaluations cost and 
whether they were satisfied with what they got for their money. 

Although a dollar amount cannot be specified, it is possible to 
describe the kinds of information you can obtain from evaluations 
at different cost levels. Think of the process of building a house. If 
you spend a small amount of money, you can build the foundation 
for the house. Additional money will be required to frame the 
house and still more money will be needed to put on the roof. To 
finish the inside of the house so that it is inhabitable will require 
even more money. 

Evaluation is similar. Some general guidelines follow on what you 
may be able to get at different evaluation cost levels. 
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Lowest cost evaluations. If you spend only a minimal amount of 
money, you will be able to obtain numerical counts of participants, 
services, or products and information about the characteristics of 
participants. You also may be able to find out how satisfied 
participants were with the services or the training. But this is only 
the foundation for an evaluation. This information will not tell you 
whether you have been successful in attaining your participant 
outcome objectives. Also, at this cost level you will not have in- 
depth information about program implementation and operations 
to understand whether your program was implemented as intended 
and, if not, what changes were made and why they were made. 

Low-moderate cost evaluations. If you increase your evaluation 
budget slightly, you will be able to assess whether there has been a 
change in your participants' knowledge, attitudes, or behaviors, and 
also collect in-depth information about your program's 
implementation. However, this is only the framework of an 
evaluation. At this cost level, you may not be able to attribute 
participant changes specifically to your program because you will 
not have similar information on a comparison or control group. 

Moderate-high cost evaluations. Adding more money to your 
evaluation budget will allow you to use a comparison or control 
group, and therefore be able to attribute any changes in 
participants to the program itself. At this cost level, however, your 
information on participant outcomes may be limited to short-term 
changes-those that occurred during or immediately after 
participation in the program. 

Highest cost evaluations. At the highest cost level, you will be 
able to obtain all of the information available from the other cost 
options as well as longer term outcome information on program 
participants. The high cost of this type of evaluation is due to the 
necessity of tracking or contacting program participants after they 
have left the program. Although follow up activities often are 
expensive, longer term outcome information is important because 
it assesses whether the changes in knowledge, attitudes, or 
behaviors that your participants experienced initially are 
maintained over time. 

Basically, as you increase your budget for an evaluation, you gain a 
corresponding increase in knowledge about your success in 
attaining your program objectives. In many situations, the lowest 
cost evaluations may not be worth the expense, and, to be realistic, 
the highest cost evaluations may be beyond the scope of most 
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agencies' financial resources. As a general rule, the more money 
you are willing to invest in an evaluation, the more useful the 
information that you will obtain about your program's 
effectiveness will be, and the more useful these results will be in 
helping you advocate for your program. 
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Who Should 
Conduct Your 
Evaluation? 


IN THIS CHAPTER... 
Staffing an evaluation 
Selecting teams 
Looking at resources 


O ne decision that must be made before you begin your 
evaluation is who will conduct it. Evaluation is best 
thought of as a team effort. Although one individual 
heads the team and has primary responsibility for the 
project, this person will need assistance and cooperation from 
others. Again, think of building a house. You may hire a contractor 
to build your house, but you would not expect this professional to 
do the job alone. You know that to build your house the 
contractor will need guidance from you and assistance from a 
variety of technical experts including an architect, electrician, 
plumber, carpenter, roofer, and mechanical engineer. 


Similarly, in conducting an evaluation, the team leader will need 
assistance from a variety of individuals in determining the focus 
and design of the evaluation, developing the evaluation plan and 
sampling plan (if necessary), constructing data collection 
instruments, collecting the evaluation data, analyzing and 
interpreting the data, and preparing the final report. 


WHAT ARE SOME POSSIBLE TYPES OF EVALUATION 

TEAMS? 

There are many types of evaluation teams that you could assemble. 
Three possible options for evaluation teams follow: 


An outside evaluator (which may be an individual, research 
institute, or consulting firm) who serves as the team leader and 
is supported by in-house staff (Team 1). 
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An in-house evaluator who serves as the team leader and is 
supported by program staff and an outside consultant (Team 
2 ). 


EVALUATION TEAM 
OPTIONS 

Team 1: Outside evaluator 
supported by program staff. 
Team 2: In-house evaluation 
team supported by consultant 
and program staff. 

Team 3: In-house evaluation 
team supported by program staff. 


■ An in-house evaluator who serves as the team leader and is 
supported by program staff (Team 3). 

Whatever team option you select, you must make sure that you, 
the program manager, are part of the team. Even if your role is 
limited to one of overall evaluation management, you must 
participate in all phases of the evaluation effort. 

Team 1 option: Am outside evaluator with support from 
program staff 
Possible advantages: 

■ Because outside evaluators do not have a stake in the 
evaluation's findings, the results may be perceived by current 
or potential funders as more objective. 

■ Outside evaluators may have greater expertise and knowledge 
than agency staff about the technical aspects involved in 
conducting an evaluation. 

■ Outside evaluators may offer a new perspective to program 
operations 

■ The evaluation may be conducted more efficiently if the 
evaluator is experienced. 

Possible disadvantages: 

■ Hiring an outside evaluator can be expensive. 

■ Outside evaluators may not have an adequate understanding of 
the issues relevant to your program or target population. 

Selecting this team does not mean that you or your staff need not 
be involved in the evaluation. You and other staff members must 
educate the evaluator about the program, participants, and 
community. Other staff or advisory board members must also be 
involved in planning the evaluation to ensure that it addresses your 
program's objectives and is appropriate for your program's 
participants. 

When deciding on your option, keep in mind that although hiring 
an outside evaluator to conduct an evaluation may appear to be 
expensive, ultimately it may be less expensive than channeling staff 
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WHO SHOULD CONDUCT YOUR EVALUATION? 


If you choose to use Team 3, an 
in-house evaluation team, it is 
important for you to define 
specific roles and responsibilities 
for the evaluation team and your 
staff to strengthen the objectivity 
of your evaluation. 


resources into an evaluation that is not correctly designed or 
implemented. 

Team 2 option: In-house evaluation team leader with support 
from program staff and an outside consultant 
Possible advantages: 

■ An evaluation team headed by an in-house staff member may 
be less expensive than hiring an outside evaluator (this is not 
always true). 

■ The use of an agency staff member as a team leader may 
increase the likelihood that the evaluation will be consistent 
with program objectives. 

Possible disadvantages: 

■ The greater time commitment required of staff may outweigh 
the cost reduction of using the outside professional as a 
consultant instead of a team leader. 

■ A professional evaluator used only for consulting purposes 
may not give as much attention to the evaluation tasks as may 
be needed. Like the Team 3 option, Team 2 may be perceived 
as less objective than using an outside evaluator. 

This second option is a good choice if you feel that you have 
sufficient staff resources to implement the evaluation, but need 
assistance with the technical aspects. An evaluation consultant, for 
example, may help with developing the evaluation design, 
conducting the data analyses, or selecting or constmcting 
appropriate data collection tools. You will also want the consultant 
to help you develop the evaluation plan to ensure that it is 
technically correct and that what you plan to do in the evaluation 
will allow you to answer your evaluation questions. 

Team 3 option: In-house evaluation team leader with support 
from program and other agency staff 
Possible advantages: 

An in-house evaluation team may be the least expensive option, 
but this is not always tme. An in-house staff evaluation team 
promotes maximum involvement and participation of program 
staff and can contribute to building staff expertise for future 
evaluation efforts. 
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REMINDER: Selecting the best 
evaluation team depends on your 
program’s funding requirements 
and your agency’s resources and 
capabilities. 


Possible disadvantages: 

■ An in-house team may not be sufficiently knowledgeable or 
experienced to design and implement the evaluation. 

■ Potential funders may not perceive evaluation results as 
objective. 

This option presumably avoids the expense of hiring an outside 
professional, so it is generally thought to be less costly than other 
evaluation teams. However, because it requires a greater 
commitment of staff time, you may discover that it is just as costly 
as using an outside evaluator either as a team leader or as a 
consultant. You may want to conduct a careful analysis of staff 
time costs compared to outside consultant costs before you decide 
on this team option. 

HOW CAN YOU DECIDE WHAT TEAM IS BEST FOR 

YOU? 

Before you decide on the best team to assemble, you will need to 
consider two important issues. 

Your program's funding requirements. Often a funding agency 
requires that you hire an outside evaluator to conduct your 
evaluation. This type of evaluator is often referred to as a third- 
party evaluator and is someone who is not affiliated with your 
agency in any way - someone with evaluation experience who will 
be objective when evaluating your program. 

Your program's resources and capabilities. You can assemble 
different types of teams depending on your agency's resources and 
how you will use the findings. To determine what internal 
resources are available, examine your staff s skills and experience in 
planning an evaluation, designing data collection procedures and 
instruments, and collecting and analyzing data and information. 

Also, examine the information you already have available through 
program activities. If, for example, you collect and review 
information from the Runaway and Homeless Youth Management 
Information System or the Head Start Program Information 
Report (or any other organized participant database or information 
system), you may be able to use this information as evaluation data. 

If you conduct entrance and exit interviews of participants or 
complete paperwork or logs on participants' progress in the 
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WHO SHOULD CONDUCT YOUR EVALUATION? 


program, this information may also be used as part of an 
evaluation. 

The checklist below can help you decide what type of team you 
may need. Answer the questions based on what you know about 
your resources. 


Resources for Appropriate Team Selection (check one) 

Yes 

No 

1. Does your agency or program have funds designated for evaluation 
purposes? 



2. Have you successfully conducted previous evaluations of similar programs, 
components, or services? 



3. Are existing program practices and information collection forms useful for 
evaluation purposes? 



4. Can you collect evaluation information as part of your regular program 
operations (at intake, termination)? 



5. Are there agency staff who have training and experience in evaluation- 
related tasks? 



6. Are there advisory board members who have training and experience in 
evaluation-related tasks? 




Whatever team you select, remember that you and your staff need 
to work with the evaluation team and be involved in all evaluation 
planning and activities. Your knowledge and experience working 
with program participants and the community are essential for an 
evaluation that will benefit the program, program participants, 
community, and funders. 

The checklist above can help you select your evaluation team in 
the following ways: 

S If your answer to all the resource questions is "no," you may 
want to consider postponing your evaluation until you can 
obtain funds to hire an outside evaluator, at least on a 
consultancy basis. You may also want to consider budgeting 
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funds for evaluation purposes in your future program planning 
efforts. 

If your answer to question 1 is "yes," but you answer "no" to 
all other questions, you will need maximum assistance in 
conducting your evaluation and Team 1 (an outside evaluator 
with in-house support) is probably your best choice. 

S If you answer "no" to question 1, but "yes" to most of the 
other resource questions, then Team 3 (in-house staff only) 
may be an appropriate choice for you. Keep in mind, however, 
that if you plan to use evaluation findings to seek program 
funding, you may want to consider using the Team 2 option 
(in-house evaluation team with outside consultant) instead and 
trying to obtain evaluation funds from other areas of your 
agency's budget. 

S If your answer to question 1 is "yes" and the remainder of your 
answers are mixed (some "yes" and some "no") then either the 
Team 1 or Team 2 option should be effective. 

The next chapter provides advice on how to locate, select, hire, 
and manage an outside evaluator or consultant. This information 
will be particularly helpful in assembling Teams 1 or 2. If you plan 
to conduct the evaluation using the Team 3 option, Chapter 4 may 
still be useful, because it provides suggestions on locating 
resources that may assist you in your evaluation efforts. 
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How Do You Hire 
and Manage an 
Outside 
Evaluator? 


IN THIS CHAPTER... 

Locating an evaluator 
Selecting a candidate 
Preparing a contract 
Handling problems along the way 


C areful selection of an outside evaluator can mean the 
difference between a positive and a negative experience. 
You will experience the maximum benefits from an 
evaluation if you hire an evaluator who is willing to work 
with you and your staff to help you better understand your 
program, learn what works, and discover what program 
components may need refining. If you build a good relationship 
with your evaluator you can work together to ensure that the 
evaluation remains on track and provides the information you and 
your funding agency want. 

FINDING AN OUTSIDE EVALUATOR 

There are four basic steps for finding an evaluator. These steps are 
similar to any you would use to recmit and hire new program staff. 
Public agencies may need to use a somewhat different process and 
involve other divisions of the agency. If you are managing a 
program in a public agency, check with your procurement 
department for information on regulations for hiring outside 
evaluators or consultants. 

Step 1: Develop a job description. The first step in the hiring 
process is to develop a job description that lists the materials, 
services, and products to be provided by the evaluator. In 
developing your job description, you will need to know the types 
of evaluation activities you want this person to perform and the 
time lines involved. Evaluator responsibilities can involve 
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BASIC STEPS FOR FINDING 
AND HIRING AND 
EVALUATOR 

Develop a job description 
Locate sources for evaluators 
Advertise 

Interview candidates 
Write a contract 


developing an evaluation plan, providing progress reports, 
developing data collection instruments and forms, collecting and 
analyzing data, and writing reports. If you think you need 
assistance in developing a job description, ask another agency that 
has experience in hiring outside evaluators for help. Advisory 
board members may also be able to assist with this task. 

Step 2: Locate sources for evaluators. Potential sources useful 
for finding an evaluator include the following: 

Other agencies that have used outside evaluators. Agencies in 
your community that are like yours are a good source of 
information about potential outside evaluators. These agencies may 
be able to recommend a good evaluator, suggest methods of 
advertising, and provide other useful information. This is one of 
the best ways to find an evaluator who understands your program 
and is sensitive to the community you serve. 

Evaluation divisions of State or local agencies. Most State or 
local government agencies have planning and evaluation 
departments. You may be able to use individuals from these 
agencies to work with you on your evaluation. Some evaluation 
divisions are able to offer their services at no cost as an "in-kind" 
service. If they are unable to respond to a request for proposal or 
provide you with in-kind services, staff members from these 
divisions may be able to direct you toward other organizations that 
are interested in conducting outside evaluations. 

Local colleges and universities. Departments of sociology, 
psychology, social work/ social welfare, education, public health, 
and public administration, and university-based research centers 
are possible sources within colleges and universities. Well-known 
researchers affiliated with these institutions may be readily 
identifiable. If they cannot personally assist you, they may be able 
to refer you to other individuals interested in performing local 
program evaluations. 

Technical assistance providers. Some Federal grant programs 
include a national or local technical assistance provider. If your 
agency is participating in this kind of grant program, assistance in 
identifying and selecting an evaluator is an appropriate technical 
assistance request. 

The public library. Reference librarians may be able to direct you 
to new sources. They can help identify local research firms and 
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ELEMENTS OF AN 

EFFECTIVE 

ADVERTISEMENT 

Y Your agency’s name, address, 
and telephone number, and a 
contact person (optional, but if 
you include one, this person 
should be ready to handle 
inquiries). 

V Brief description of program, 
including objectives, types of 
evaluation anticipated, available 
budget, and period of 
performance (for both program 
and evaluation). 

V Principal tasks of the evaluator. 

V Required evidence of expertise 
(such as letters of introduction, a 
resume, a list of references, or a 
description of a recent 
evaluation). 

Y Whether an interview is 
required (strongly re-commended 
for candidates under 
consideration). 

Y Other requirements (such as 
whether you will accept a faxed 
or e-mailed application) and the 
deadline for a response to the 
advertisement. 

[Note that if you are a public 
agency, you may have other 
restrictions related to 
procurement.] 


may be able to provide you with conference proceedings that list 
program evaluators who were presenters. 

Research institutes and consulting firms. Many experienced 
evaluators are part of research institutes and consulting firms. They 
are sometimes listed in the yellow pages under "Research" or 
"Marketing Research." They also can be located by contacting your 
State human services departments to get a listing of the firms that 
have bid on recent contracts for evaluations of State programs. 

National advocacy groups and local foundations, such as The 
United Way, American Public Welfare Association, Child 
Welfare League of America, and the Urban League. The staff 
and board members of these organizations may be able to provide 
you with names of local evaluators. They may also be able to offer 
insight on evaluations that were done well or evaluators especially 
suited to your needs. 

Professional associations, such as the American Evaluation 
Association, American Sociological Association, and the 
Society for Research on Child Development. Many evaluators 
belong to the American Evaluation Association. These 
organizations can provide you with a list of members in your area 
for a fee and may have tips on how you should advertise to attract 
an evaluator that best meets your needs. Additional information on 
these organizations is provided in the appendix. 

Step 3: Advertise and solicit applications. After you have 
developed a job description, identified possible sources for 
evaluators, and found ways to advertise the position, you are ready 
to post an advertisement to get applications. Advertising in the 
local paper, posting the position at a local college or university, or 
working with your local government's human resource department 
(if you are a public agency) are possible ways of soliciting 
applications. Agency newsletters, local and national meetings, and 
professional journals are additional sources where you can post 
your advertisement. 

It is wise to advertise as widely as possible, particularly if you are in 
a small community or are undertaking an evaluation for the first 
time. Several advertising sources will ensure that you receive 
multiple responses. You should build in as much time as possible 
between when you post the position and when you plan to review 
applications. 
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PROXIMITY OF 
EVALUATORS 

When considering an evaluator 
who is not local, you need to 
balance the issues of cost, 
availability, familiarity with your 
program (or other similar 
programs) and commitment. 
Although distance from the 
program is sometimes a barrier, 
many programs have successfully 
worked with out-of-town 
evaluators. 


If you have sufficient time, you may want to consider a two-step 
process for applications. The position would still be advertised, but 
you would send evaluators who respond to your advertisement 
more detailed information about your evaluation requirements and 
request a description of their approach. For example, you could 
send potential evaluators a brief description of the program and 
the evaluation questions you want to answer, along with a 
description of the community you serve. This would give them an 
opportunity to propose a plan that more closely corresponds to 
your program needs. 

Step 4: Review applications and interview potential 
candidates. In reviewing applications, consider the candidate's 
writing style, type of evaluation plan proposed, language (jargon 
free), experience working with your type of program and staff, 
familiarity with the subject area of your program, experience 
conducting similar evaluations, and proposed costs. 

After you have narrowed your selection to two or three candidates, 
you are ready to schedule an in-person interview. This interview 
will give you the opportunity to determine whether you and the 
evaluator are compatible. As you do for other job applicants, you 
will need to check references from other programs that worked 
with your candidate. 

WHAT TO DO WHEN YOU HAVE TROUBLE HIRING 
AN EVALUATOR 

Despite your best efforts, you may encounter difficulties in hiring 
an outside evaluator, including the following: 

Few or no responses to your advertisement. Many programs, 
particularly ones in isolated areas, have struggled to obtain even a 
few responses to their advertisements. Check with your Federal 
Project Officer to find out whether he or she can offer you 
suggestions, consult with other programs in your community, and 
check with your local State or county social service agency to 
obtain advice. Your advisory board may also be useful in 
identifying potential evaluators. Another source may be an 
organization that offers technical assistance to programs similar to 
yours. 

None of the applicants is compatible with program 
philosophy and staff. If applicants do not match program needs, 
you may find it helpful to network with other programs and 
agencies in your State to learn about evaluators that agencies like 
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A GOOD EVALUATOR... 

... is willing to work 
collaboratively to develop an 
evaluation plan that meets your 
needs. 

... is able to communicate in 
simple, practical terms. 

... has experience evaluating 
similar human service delivery 
pro-grams and working with 
similar resource levels. 

... has experience with statistical 
methods. 

... has the time available to do the 
evaluation. 

... has experience developing data 
collection forms or using 
standardized instmments. 

... will work with a national 
evaluation team (if one exists). 

... will treat data confidentially. 

... can come from outside the 
participant community because 
the evaluator will be able to learn 
about, respect, and work with 
your participants and their 
multiple cultures. 

... is located near your program’s 
operations or is willing to travel 
frequently to maintain contact 
with the pro-gram (although 
travel expenses will increase the 
cost of your evaluation). 


yours have used. A compatible philosophy and approach is most 
important — tradeoffs with proximity to the evaluator may need 
to be made to find the right evaluator. 

The outside evaluator's proposed costs are higher than your 
budgeted amount. In this instance, you will need to generate 
additional funds for the evaluation or negotiate with your evaluator 
to donate some of their services (in-kind services). 

Another option is to negotiate with a university professor to 
supervise advanced degree students to conduct some of the 
evaluation activities. Information about participants and programs 
is a valuable resource, providing confidentiality is respected. For 
example, you can allow a university professor to have access to 
program information and possibly to other evaluation records in 
exchange for evaluation services such as instrument development 
or data analysis. 

MANAGING AN EVALUATION HEADED BY AN 
OUTSIDE EVALUATOR 

Often, when the decision is made to hire an outside evaluator, 
program managers and staff believe that the evaluation is "out of 
their hands." This is not true. An outside evaluator cannot do the 
job effectively without the cooperation and assistance of program 
managers and staff. 

An evaluation is like any activity taking place within your agency 
— it needs to be managed. Program managers must manage the 
evaluation just as program operations are managed. What would 
happen if your staff stopped interviewing new participants? How 
long would it be before you knew this had happened? How long 
would it be before you took action? How involved would you be in 
finding a solution? An evaluation needs to be treated with the same 
level of priority. 


CREATING A CONTRACT 

A major step in managing an evaluation is the development of a 
contract with your outside evaluator. It is important that your 
contract include the following: 

Who "owns" the evaluation information. It is important to 
specify who has ownership and to whom the information can be 
given. Release of information to outside parties should always be 
cleared with appropriate agency staff. 
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ELEMENTS OF A BASIC 
CONTRACT 

■ Agency name and address 

■ Evaluator name, address, 
affiliation (if any), and social 
security number 

■ Person from agency 
responsible for monitoring 
the contract and work 

■ Key reports, products, and 
due dates 

■ Duties and responsibilities 

■ Required meetings, including 
attendance at grantee 
conferences 

■ Payment amount, payment 
schedule, and requirements 
for payment (fees and 
expenses) 

■ Liability release 

■ Materials provided to the 
evaluator by the agency 

■ Any special terms or 
conditions, including those 
for terminating the contract 

■ Data ownership and 
publication rights 

■ Confidentiality of data 

■ Any restrictions on 
publishing evaluation results 

■ Signature and date for both 
agency representative and 
evaluator 


Any plans for publishing the evaluation results should be discussed 
and cleared before articles are written and submitted for 
publication. It is important to review publication restrictions from 
the funding agency. In some instances, the funding agency may 
have requirements about the use of data and the release of reports. 

Who will perform evaluation tasks. The contract should clarify 
who is to perform the evaluation tasks and the level of contact 
between the evaluator and the program. Some program managers 
have found that outside evaluators, after they are hired, delegate 
many of their responsibilities to less experienced staff and have 
little contact with the program managers or staff. To some extent, 
a contract can protect your program from this type of situation. 

If this problem occurs even after specification of tasks, you may 
want to talk with the senior evaluator you originally hired to offer 
the option of renegotiating his or her role. The resolution should 
be mutually agreeable to program staff and the evaluator and not 
compromise the integrity of the evaluation or program. The 
contract should specify the responsibilities of program staff as well 
as the evaluator. These responsibilities may vary depending on the 
structure of your evaluation and the amount of money you have 
available. The exhibits at the end of this chapter provide some 
guidelines on roles and responsibilities. 

Your expectations about the contact between the evaluator 
and program staff. It is very important for an outside evaluator to 
keep program staff informed about the status of the evaluation and 
to integrate the evaluation into ongoing program operations. 
Failure to do this shortchanges program staff and denies the 
program an opportunity to make important changes on an ongoing 
basis. The contract could specify attendance at staff meetings and 
ongoing reporting requirements. Setting up regular meetings, 
inviting evaluators to program events and staff meetings, and 
requiring periodic reports may help solidify the relationship 
between the program and the evaluation. Other approaches that 
may help include asking a more senior agency staff member to 
become involved with the evaluation process or withholding 
payment if the evaluator fails to perform assigned tasks. 
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DO YOU NEED OUTSIDE 
APPROVAL TO HIRE AN 
EVALUATOR? 

In most agencies, executive 
director, advisory board, or 
agency board approval is needed 
before you can extend a contract 
to your selected evaluator. In 
Native- American communities, 
the tribal council may need to 
approve selection of the 
evaluator. Check your grant 
requirements or with your 
funding agency to determine 
whether you need Federal 
approval of your evaluator. 


WHAT TO DO IF PROBLEMS ARISE 

Even with the best contract, problems can arise during the course 
of the evaluation process. These problems include the following: 

Evaluation approaches differ (the program and evaluator do 
not see eye-to-eye). Try to reach a common ground where both 
programmatic and evaluation constraints and needs are met. If 
many reasonable attempts to resolve differences have been tried 
and severe conflicts still remain that could jeopardize the program 
or the evaluation, program staff should consider terminating the 
evaluation contract. This decision should be weighed carefully and 
discussed with your funder, as a new evaluator will need to be 
recmited and brought up to speed midstream. In some situations, 
finding a new evaluator may be the best option. Before making this 
decision, however, you will need to discuss this with your program 
funders, particularly if they are providing financial support for the 
evaluation. 

Evaluation of the program requires analysis s kill s outside 
your original plan. You may find that your evaluator is in 
agreement with your assessment and is willing to add another 
person to the evaluation team who has expertise and skills needed 
to undertake additional or different analyses. Many times additional 
expertise can be added to the evaluation team by using a few hours 
of a consultant's time. Programmers, statisticians, and the like can 
augment the evaluation team without fundamentally changing the 
evaluation team's structure. 

The evaluator leaves, terminates the contract, or does not 
meet contractual requirements. If the evaluator leaves the area 
or terminates the contract, you will most likely be faced with 
recmiting a new one. In some instances, programs have 
successfully maintained their ties to evaluators who have left the 
area, but this is often difficult. When your evaluator does not meet 
contractual requirements and efforts to resolve the dispute have 
failed, public agencies should turn the case over to their 
procurement office and private agencies should seek legal counsel. 

The evaluator is not culturally competent or does not have 
any experience working with your community and the 
participants. It is not always possible to locate an evaluator with 
both experience in the type of evaluation that you need and 
experience working with specific groups and subgroups in the 
community. If your evaluator does not have experience working 
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The most important message in 
this manual is that the 
effectiveness of an evaluation 
depends on having program 
managers and staff participate as 
full partners in the evaluation, 
whether it is being conducted by 
an outside evaluator or by in- 
house staff. 


with the particular group reached by the program, you must 
educate this person about the culture (or cultures) of the 
participants' community and how it might affect the evaluation 
design, instruments, and procedures. The evaluator may need to 
conduct focus groups or interviews with community members to 
make sure that evaluation questions and activities are both 
understood by and respectful of community members. 

You are not happy with the evaluator's findings. Sometimes 
program managers and staff discover that the evaluator's findings 
are not consistent with their impressions of the program's 
effectiveness with participants. Program staff believes that 
participants are demonstrating the expected changes in behavior, 
knowledge, or attitudes, but the evaluation results do not indicate 
this. In this situation, you may want to work with your evaluator to 
make sure the instruments being used are measuring the changes 
you have been observing in the program participants. Also, 
remember that your evaluator will continue to need input from 
program staff in interpreting evaluation findings. 

You may also want your evaluator to assess whether some of your 
participants are changing and whether there are any common 
characteristics shared by participants that are or are not 
demonstrating changes. However, be prepared to accept findings 
that may not support your perceptions. Not every program will 
work the way it was intended to, and you may need to make some 
program changes based on your findings. 
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POTENTIAL RESPONSIBILITIES OF THE EVALUATOR 

• Develop an evaluation plan, in conjunction with program staff. 

• Provide monthly or quarterly progress reports to staff (written or in person). 

• Train project staff. Training topics could include: 

Using evaluation instruments, information collection activities, participant/case selection for sampling 
purposes, and other activities. 

Designing information collection instruments or selecting standardized instruments or inventories. 

• Implement information collection procedures such as: 

Interview project staff. 

Interview coordinating/collaborating agency staff. 

Interview program participants. 

Conduct focus groups. 

Observe service delivery activities. 

Review participant case records 
Develop database. 

Code, enter, and clean data. 

Analyze data. 

• Establish and oversee procedures ensuring confidentiality during all phases of the evaluation. 

• Write interim (quarterly, biannual, yearly) evaluation reports and the final evaluation report. 

• Attend project staff meetings, advisory board or interagency coordinating committee meetings, and 
grantee meetings sponsored by funding agency. 

• Present findings at local and national meetings and conferences. 


27 



THE PROGRAM MANAGER’S GUIDE TO EVALUATION 


POTENTIAL RESPONSIBILITIES OF THE PROGRAM MANAGER 

• Educate the outside evaluator about the program's operations and objectives, characteristics of the 
participant population, and the benefits that program staff expects from the evaluation. This may 
involve alerting evaluators to sensitive situations (for example, the need to report suspected child 
abuse) they may encounter during the course of their evaluation activities. 

• Provide feedback to the evaluator on whether instruments are appropriate for the target population and 
provide input during the evaluation plan phase. 

• Keep the outside evaluator informed about changes in the program's operations. 

• Specify information the evaluator should include in the report. 

• Assist in interpreting evaluation findings. 

• Provide information to all staff about the evaluation process. 

• Monitor the evaluation contract and completion of work products (such as reports). 

• Ensure that program staff is fulfilling their responsibilities (such as data collection). 

• Supervise in-house evaluation activities, such as completion of data collection instruments, and data 
entry. 

• Serve as a troubleshooter for the evaluation process, resolving problems or locating a higher level 
person in the agency who can help. 

• Request a debriefing from the evaluator at various times during the evaluation and at its conclusion. 


28 




How Do You 
Prepare for an 
Evaluation? 

W hen you build a house, you start by laying the 
foundation. If your foundation is not well constructed, 
your house will eventually develop cracks and you will 
be constantly patching them up. Preparing for an 
evaluation is like laying a foundation for a house. The effectiveness 
of an evaluation ultimately depends on how well you have planned 
it. 

Begin preparing for the evaluation when you are planning the 
program, component, or service that you want to evaluate. This 
approach will ensure that the evaluation reflects the program's 
goals and objectives. The process of preparing for an evaluation 
should involve the outside evaluator or consultant (if you decide to 
hire one), all program staff who are to be part of the evaluation 
team, and anyone else in the agency who will be involved. The 
following steps are designed to help you build a strong foundation 
for your evaluation. 


IN THIS CHAPTER... 

Preparing for an evaluation 
Program models 

Working with assumptions about 
your program 

Providing a context for your 
evaluation 


STEP 1: DECIDE WHAT TO EVALUATE 

Programs vary in size and scope. Some programs have multiple 
components, whereas others have only one or two. You can 
evaluate your entire program, one or two program components, or 
even one or two services or activities within a component. To a 
large extent, your decision about what to evaluate will depend on 
your available financial and staff resources. If your resources are 
limited, you may want to narrow the scope of your evaluation. It is 
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STEPS TO PREPARE FOR AN 
EVALUATION 

Step 1: Decide what to evaluate 
Step 2: Build a model of your 
program 

Step 3: State objectives in 
measurable terms 
Step 4: Identify the context for 
your evaluation 


better to conduct an effective evaluation of a single program 
component than to attempt an evaluation of several components 
or an entire program without sufficient resources. 

Sometimes the decision about what to evaluate is made for you. 
This often occurs when funders require evaluation as a condition 
of a grant award. Funders may require evaluations of different 
types of programs including, but not limited to, demonstration 
projects. Evaluation of demonstration projects is particularly 
important to funders because the purpose of these projects is to 
develop and test effective program approaches and models. 

At other times, you or your agency administrators will make the 
decision about what to evaluate. As a general rule, if you are 
planning to implement new programs, components, or services, 
you should also plan to evaluate them. This step will help you 
determine at the outset whether your new efforts are implemented 
successfully, and are effective in attaining expected participant 
outcomes. It will also help identify areas for improvement. 

If your program is already operational, you may decide you want to 
evaluate a particular service or component because you are unsure 
about its effectiveness with some of your participants. Or, you may 
want to evaluate your program because you believe it is effective 
and you want to obtain additional funding to continue or expand 
it. 

STEP 2: BUILD A LOGIC MODEL OF YOUR PROGRAM 

Whether you decide to evaluate an entire program, a single 
component, or a single service, you will need to build a logic model 
that clearly describes what you plan to do. A logic model will 
provide a stmctural framework for your evaluation. You will need 
to develop a clear picture of the particular program, component, or 
service to be evaluated so that everyone involved has a shared 
understanding of what they are evaluating. Building a logic model 
will help you with this task. 

There are a variety of types of logic models. There are also a 
number of different terms for logic models, including program 
model, program theory, and theory of change. In general, the logic 
model represents a series of logically related assumptions about the 
program's participant population and the changes you hope to 
bring about in that population as a result of your program. It 
should reflect both how the program will work and how it will 
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ASSUMPTIONS are the 
underlying principles that explain 
why the program’s component or 
service was developed. 


have its effect on the target population. Many examples can be 
found in the resources listed in the appendix. 

The model discussed in this chapter focuses on the program's 
implementation and participant outcome objectives. A sample 
completed program model and a worksheet that can be used to 
develop a model for your program appear at the end of this 
chapter. The program model includes the following features. 

Assumptions about your target population. Your assumptions 
about your target population are the reasons why you decided to 
develop a program, program component, or service. These 
assumptions may be based on theory, your own experiences in 
working with the target population, or your review of existing 
research or program literature. 

Using the worksheet, you would write your assumptions in column 
1. Some examples of assumptions about a participant population 
that could underlie development of a program and potential 
responses to these assumptions include the following: 

Assumption: Children of parents who abuse alcohol or other 
dmgs are at high risk for parental abuse or neglect. 

»Response: Develop a program to work with families to address 
substance abuse and child abuse problems simultaneously. 

Assumption: Runaways and homeless youth are at high risk for 
abuse of alcohol and other dmgs. 

» Response . Develop a program that provides drug abuse 
intervention or prevention services to runaway and homeless 
youth. 

Assumption: Families with multiple interpersonal, social, and 
economic problems need early intervention to prevent the 
development of child maltreatment, family violence, alcohol and 
other drug (AOD) problems, or all three. 

» Response: Develop an early intervention program that provides 
comprehensive support services to at-risk families. 

Assumption. Children from low-income families are at high risk 
for developmental, educational, and social problems. 
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»Response: Develop a program that enhances the developmental, 
educational, and social adjustment opportunities for children. 

Assumption: Child protective services (CPS) workers do not have 
sufficient skills for working with families in which substance abuse 
and child maltreatment coexist. 

»Response: Develop a training program that will expand the 
knowledge and skill base of CPS workers. 

Program interventions (implementation objectives). The 

program's interventions or implementation objectives represent 
what you plan to do to respond to the problems identified in your 
assumptions. They include the specific services, activities, or 
products you plan to develop or implement. Using the worksheet, 
you can fill in your program implementation objectives in column 
2. Some examples of implementation objectives that correspond to 
the above assumptions include the following: 

■ Provide intensive in-home services to parents and children. 

■ Provide drug abuse education services to runaway and homeless youth. 

■ Provide in-home counseling and case management services to low-income 
mothers with infants. 

■ Provide comprehensive child development services to children and families. 

■ Provide multidisciplinary training to CPS workers. 

Implementation objectives are also sometimes known as 
“outputs,” which should be distinguished from “outcomes.” 

Immediate outcomes (immediate participant outcome 
objectives). Immediate participant outcome objectives can be 
entered in column 3. These are your expectations about the 
changes in participants' knowledge, attitudes, and behaviors that 
you expect to result from your intervention by the time 
participants complete the program. Examples of immediate 
outcomes linked to the above interventions include the following: 

■ Parents will acknowledge their substance abuse problems. 

■ Youth ivill demonstrate changes in their attitudes toward use of alcohol 
and other drugs. 
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■ Mothers will increase their knowledge of infant development and of effective 
and appropriate parenting practices. 

■ Children will demonstrate improvements in their cognitive and 
interpersonal functioning 

■ CPS workers will increase their knowledge about the relationship between 
substance abuse and child maltreatment and about the appropriate service 
approach for substance-abusing parents. 

Whereas “outputs” refer to the program services or activities, 
“outcomes” refer to participant knowledge, attitudes, or behaviors. 

Intermediate outcomes. Intermediate outcomes, entered in 
column 4, represent the changes in participants that you think will 
follow after immediate outcomes are achieved. Examples of 
intermediate outcomes include the following: 

■ After parents acknowledge their AOD abuse problems, they will seek 
treatment to address this problem. 

■ After parents receive treatment for AOD abuse, there will be a 
reduction in the incidence of child maltreatment. 

■ After runaway and homeless youth change their attitudes toward AOD 

use , they will reduce this use. 

■ After mothers have a greater understanding of child development and 
appropriate parenting practices, they will improve their parenting 
practices with their infants. 

■ After children demonstrate improvements in their cognitive and 
interpersonal functioning they will increase their ability to 
function at an age-appropriate level in a particular setting. 

■ After CPS workers increase their knowledge about working with families 
in which AOD abuse and child maltreatment coexist, they will 
improve their s k ills for working with these families. 

Long-term outcomes. Long-term outcomes, specified in the last 
column of the model, represent your expectations about the long- 
term effects of your program on participants or the community. 
They are derived logically from your immediate and intermediate 
outcomes. Some people call these distal outcomes, since they may 
refer to outcomes that are less closely related to the program 
activities or to outcomes that are more distant in the future. In 
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REMEMBER TO STATE 
YOUR OBJECTIVES IN 
MEASURABLE TERMS. 
Whatever objectives you choose 
to focus on, make sure you have 
stated these objectives in a way 
that allows them to be measured. 
This is the most important step in 
preparing for your evaluation. All 
subsequent decisions, including 
the evaluation designs, data 
collection, data analyses, and 
reports, will be based on this step. 


contrast, immediate and intermediate outcomes can be referred to 
as proximal outcomes. Examples of long-term outcomes include 
the following: 

■ After runaway and homeless youth reduce their AOD abuse, they will 
seek services designed to help them resolve other 
problems they may have. 

■ After mothers of infants become more effective parents, the need for 
out-of-home placements for their children will be 
reduced. 

■ After CPS workers improve their skills for working with families in 
which AOD abuse and child maltreatment coexist, collaboration and 
integration of services between the child welfare and the 
substance abuse treatment systems will increase. 

Logic models are not difficult to constmct, and they lay the 
foundation for your evaluation by clearly identifying your program 
implementation and participant outcome objectives. These models 
can then be stated in measurable terms for evaluation purposes. 

STEP 3: STATE YOUR PROGRAM IMPLEMENTATION 
AND PARTICIPANT OUTCOME OBJECTIVES IN 
MEASURABLE TERMS 

The logic model serves as a basis for identifying your program's 
implementation and participant outcome objectives. Initially, you 
should focus your evaluation on assessing whether implementation 
objectives and immediate participant outcome objectives were 
attained. This task will allow you to assess whether it is worthwhile 
to commit additional resources to evaluating attainment of 
intermediate and final or long-term outcome objectives. 

Remember, every program, component, or service can be 
characterized by two types of objectives — implementation 
objectives and outcome objectives. Both types of objectives will 
need to be stated in measurable terms. 

Often program managers believe that stating objectives in 
measurable terms means that they have to establish performance 
standards or some kind of arbitrary "measure" that the program 
must attain. This is not correct. Stating objectives in measurable 
terms simply means that you describe what you plan to do in your 
program and how you expect the participants to change in a way 
will allow you to measure these objectives. From this perspective, 
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MEASURABLE means that you 
define “how many, when, for 
how long, what type, and how 
much” in your implementation 
objectives. 


measurement can involve anything from counting the number of 
services (or determining the duration of services) to using a 
standardized test that will result in a quantifiable score. Some 
examples of stating objectives in measurable terms are provided 
below. 

Stating implementation objectives in measurable terms. 

Examples of implementation objectives include the following: 

What you plan to do — The services/ activities you plan to provide or 
the products you plan to develop, and the duration and intensity of 
the services or activities. 

Who will do it — What the staffing arrangements will be; the 
characteristics and qualifications of the program staff who will 
deliver the services, conduct the training, or develop the products; 
and how these individuals will be recmited and hired. 

Who you plan to reach and how many — A description of the 
participant population for the program; the number of participants 
to be reached during a specific time frame; and how you plan to 
recmit or reach the participants. 

These objectives are not difficult to state in measurable terms. You 
simply need to be specific about your program's operations. The 
following example demonstrates how general implementation 
objectives can be transformed into measurable objectives. 

General objective: Provide substance abuse prevention and 
intervention services to runaway youth. 

» Measurable objectives: 

What you plan to do — Provide eight dmg abuse education class 
sessions per year with each session lasting for 2 weeks and 
involving 2-hour classes convened for 5 days of each week. 

Develop a curriculum that will include at least two self-esteem 
building activities, four presentations by youth who are in recovery, 
two field trips to recreational facilities, four role playing activities 
involving parent-child interactions, and one educational lecture on 
dmgs and their effects. 

Who will do it — Classes will be conducted by two counselors. One 
will be a certified addictions counselor, and the other will have at 
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least 2 years of experience working with runaway and homeless 
youth. 

The curriculum for the classes will be developed by the 2 
counselors in conjunction with the clinical director and an outside 
consultant who is an expert in the area of AOD abuse prevention 
and intervention. 

Counselors will be recruited from current agency staff and will be 
supervised by the agency clinical director who will provide 3 hours 
of supervision each week. 

Who you plan to reach and how many — Classes will be provided to all 
youth residing in the shelter during the time of the classes (from 8 
to 14 youth for any given session) and to youth who are seeking 
crisis intervention services from the youth services agency 
(approximately 6 youth for each session). All youth will be between 
13 and 17 years old. Youth seeking crisis intervention services will 
be recmited to the classes by the intake counselors and the clinical 
director. 

A blank worksheet that can be used to state your implementation 
objectives in measurable terms is provided at the end of this 
chapter. From your description of the specific characteristics for 
each objective, the evaluation will be able to assess, on an ongoing 
basis whether the objectives were attained, the types of problems 
encountered during program implementation, and the areas where 
changes may need to be made. For example, using the example 
provided above, you may discover that the first class session 
included only two youth from the crisis intervention services. You 
will then need to assess your recmitment process, asking the 
following questions: 

How many youth sought crisis intervention services during that timeframe ? 

How many youth agreed to participate ? 

What harriers were encountered to participation in the classes (such as youth or 
parent reluctance to give permission, lack of transportation, or lack of 
interest among youth)? 

Based on your answers to these questions, you may decide to 
revise your recruitment strategies, train crisis intervention 
counselors to be more effective in recmiting youth, visit the family 
to encourage the youth's participation, or offer transportation to 
youth to make it easier for them to attend the classes. 
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Stating participant outcome objectives in measurable terms. 

This process requires you to be specific about the changes in 
knowledge, attitudes, awareness, or behavior that you expect to 
occur as a result of participation in your program. One way to be 
specific about these changes is to ask yourself the following 
question: 

How will we know that the expected changes occurred ? 

To answer this question, you will have to identify the evidence 
needed to demonstrate that your participants have changed. The 
following examples demonstrate how participant outcome 
objectives may be stated in measurable terms. A worksheet for 
defining measurable participant outcome objectives appears at the 
end of this chapter. 

General objective: We expect to improve the parenting skills 
of program participants. 

»Measurable objective: Parents participating in the program will 
demonstrate significant increases in their scores on an instrument 
that measures parenting skills from intake to completion of the 
parenting education classes. 

General objective: We expect to reduce the use of alcohol and 
other drugs by youth participating in the substance abuse 
intervention program. 

»Measurable objective: Youth will indicate significant decreases 
in their scores on an instmment that measures use of alcohol and 
other drugs from intake to after program participation. 

General objective: We expect to improve CPS workers ' ability 
to work effectively with families in which child maltreatment 
and parental substance abuse problems coexist. 

»Measurable objective: CPS workers will demonstrate significant 
increases in their scores on instruments that measure knowledge of 
substance abuse and child maltreatment issues and skills for 
working with these families from before to after training. 

General objective: We expect to reduce the risk of child 
maltreatment for children in the families served. 

»Measurable objective: Families served by the program will be 
significantly less likely than a similar group of families to be 
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reported for child maltreatment for 6 months after they complete 
the program. 

STEP 4: IDENTIFY THE CONTEXT FOR YOUR 
EVALUATION 

Part of planning for an evaluation requires understanding the 
context in which the evaluation will take place. Think again about 
building a house. Before you can design your house, you need to 
know something about your lot. If your lot is on a hill, you must 
consider the slope of the hill when you design your house. If there 
are numerous trees on the lot, you must design your house to 
accommodate the trees. 

Similarly, program evaluations do not take place in a vacuum, and 
the context of an evaluation must be considered before the 
evaluation can be planned and designed. Although many 
contextual factors can affect your evaluation, the most common 
factors pertain to your agency, your staff, and your participant 
population. 

The agency context. The characteristics of an agency 
implementing a program affects both the program and the 
evaluation. The aspects of your agency that need to be considered 
in preparing for your evaluation include the following: 

The agency's evaluation-related resources. Does the agency have a 
management information system in place that can be used to 
collect data on participants and services? Does the agency have an 
advisory board that includes members who have experience 
evaluating programs? Does the agency have discretionary funds in 
the budget that can be used for an evaluation? 

The agency's history of conducting program evaluations. Has the agency 
evaluated its programs before? If yes, was the experience a negative 
or positive one? If it was negative, what were the problems 
encountered and how can they be avoided in the current 
evaluation? Are the designs of previous agency evaluations 
appropriate for the evaluation you are currently planning? 

If the agency has a history of program evaluation, you may be able 
to use the previous evaluation designs and methodology for your 
current evaluation. Review these with your outside evaluator or 
consultant to determine whether they are applicable to your 
current needs. If they are applicable, this will save you a great deal 
of time and money. 
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CULTURE refers to the shared 
values, traditions, norms, 
customs, arts, history, institutions 
and experiences of a group of 
people. The group may be 
identified by race, age, ethnicity, 
language, national origin, religion, 
or other social category or 
grouping. 


The program's relationship to other agency activities. Is the program you 
want to evaluate integrated into other agency activities, or does it 
function as a separate entity? What are the relationships between 
the program and other agency activities? If it is integrated, how will 
you evaluate it apart from other agency activities? This can be a 
complicated process. If your evaluation team does not include 
someone who is an experienced evaluator, you may need assistance 
from an outside consultant to help you with this task. 

The staff context. The support and full participation of program 
staff in an evaluation is critical to its success. Sometimes 
evaluations are not successfully implemented because program 
staff who are responsible for data collection do not consistently 
administer or complete evaluation forms, follow the directions of 
the evaluation team, or make concerted efforts to track participants 
after they left the program. The usual reason for staff-related 
evaluation problems is that staff were not adequately prepared for 
the evaluation or given the opportunity to participate in its 
planning and development. Contextual issues relevant to program 
staff include the following: 

The staffs experiences in participating in program evaluations. Have your 
staff participated in evaluations prior to this one? If yes, was the 
experience a positive or negative one? If no, how much do they 
know about the evaluation process and how much training will 
they need to participate as full partners in the evaluation? 

If staff have had negative experiences with evaluation, you will 
need to work with them to emphasize the positive aspects of 
evaluation and to demonstrate how this evaluation will be different 
from prior ones. All staff will need careful training if they are to be 
involved in any evaluation activities, and this training should be 
reinforced throughout the duration of the evaluation. 

The staffs attitudes toward evaluation. Do your staff have positive or 
negative attitudes toward evaluation? If negative, what can be done 
to make them more positive? How can they be encouraged to 
support and participate fully in the evaluation? 

Negative attitudes sometimes can be counteracted when program 
managers demonstrate enthusiasm for the evaluation and when 
evaluation activities are integrated with program activities. It may 
be helpful to demonstrate to staff how evaluation instruments also 
can be used as assessment tools for participants and therefore help 
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GETTING PROGRAM 
PARTICIPANTS INVOLVED. 
You may find it useful to discuss 
the evaluation with program 
participants, if it seems 
reasonable to you. You could 
describe what you plan to do and 
something about how you plan to 
do it, and obtain feedback from 
participants concerning their 
attitudes and opinions about the 
evaluation. 


staff develop treatment plans or needs assessments for individual 
participants. 

The staffs knowledge about evaluation. Are your staff knowledgeable 
about the practices and procedures required for a program 
evaluation? Do any staff members have a background in 
conducting evaluations that could help you with the process? 

Staff who are knowledgeable about evaluation practices and 
procedures can be a significant asset to an evaluation. They can 
assume some of the evaluation tasks and help train and supervise 
other staff on evaluation activities. 

The participant population context. Before designing an 
evaluation, it is very important to understand the characteristics of 
your participant population. The primary issue relevant to the 
participant population context concerns the potential diversity of your 
program population. For example, is the program population similar 
or diverse with respect to age, gender, ethnicity, socioeconomic 
status, and literacy levels? If the population is diverse, how can the 
evaluation address this diversity? 

Participant diversity can present a significant challenge to an 
evaluation effort. Instmments and methods that may be 
appropriate for some participants may not be for others. For 
example, written questionnaires may be easily completed by some 
participants, but others may not have adequate literacy levels. 
Similarly, face-to-face interviews may be appropriate for some of 
the cultural groups the program serves, but not to others. 

If you serve a diverse population of participants, you may need to 
be flexible in your data collection methods. You may design an 
instrument, for example, that can be administered either as a 
written instrument or as an interview instmment. You also may 
need to have your instmments translated into different languages. 
However, it is important to remember that just translating an 
instrument does not necessarily mean that it will be culturally 
appropriate. 

If you serve a particular cultural group, you may need to select the 
individuals who are to collect the evaluation information from the 
same cultural or ethnic group as your participants. If you are 
concerned about the literacy levels of your population, you will 
need to pilot test your instmments to make sure that participants 
understand what is being asked of them. More information related 
to pilot tests appears in Chapter 7. 
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Identifying contextual issues is essential to building a solid 
foundation for your evaluation. During this process, you will want 
to involve as many members of your expected evaluation team as 
possible. The decisions you make about how to address these 
contextual issues in your evaluation will be fundamental to 
ensuring that the evaluation operates successfully and that its 
design and methodology are appropriate for your participant 
population. 

After you have completed these initial steps, it is time to "frame" 
your house. To frame a house, you need blueprints that detail the 
plans for the house. The blueprint for an evaluation is the 
evaluation plan. Chapter 6 discusses the elements that go into 
building this plan. 
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SAMPLE LOGIC MODEL 
Child Abuse Prevention Program 


Assumptions about Target 
Population 

Interventions 

(Implementation Objectives) 

Immediate Outcomes 
(Participant Outcome 
Objectives) 

Intermediate Outcomes 
(Participant Outcome 
Objectives) 

Long-Term Outcomes 

General: Children of parents 
who abuse alcohol and other 
drugs (AOD) are at high risk 
for parental maltreatment. 

General: Develop a program 
that addresses AOD abuse 
and child maltreatment 
simultaneously. 

General: Improve overall 
quality of family functioning 

General: Reduce AOD abuse 
and child maltreatment in 
families. 

Eliminate AOD abuse and 
child maltreatment among 
participating families parental 
maltreatment. 

Specific: The risk for child 
abuse will decrease if parents 
stop using alcohol and other 
drugs 

Substance abuse treatment 
services 

Home visiting services 

Parents acknowledge AOD 
abuse and seek help 

Parents reduce AOD use 

Parents complete AOD 
treatment and continue with 
support groups or other 
aftercare services 

Eliminate AOD abuse among 
participating parents 

Specific: The risk for child 
abuse will decrease if parents 
develop effective parenting 
skills. 

Parent effectiveness 
education 

Parent support groups 

Child development education 

Parents increase knowledge 
of parenting skills 

Parents increase knowledge 
of child development 

Parents acknowledge impact 
of their AOD abuse on 
children 

Parents demonstrate changes 
in their parenting practices 
with children 

Eliminate child maltreatment 
among participating parents 

Specific: Children of AOD 
abusing parents are at high 
risk for AOD abuse. 

Skill building classes 
Recreational activities 

Drug abuse education 
services 

Children increase skills in 
dealing with AOD abusing 
parents 

Children increase knowledge 
of AOD 

Children increase self-esteem 

Prevent AOD abuse among 
children in participating 
families 




LOGIC MODEL WORKSHEET 


Assumptions about Target Interventions Immediate Outcomes Intermediate Outcomes Long Term Outcomes 

Population (Implementation Objectives) (Participant Outcome (Participant Outcome 

Objectives) Objectives) 



SAMPLE WORKSHEET: IMPLEMENTATION OBJECTIVES STATED IN MEASURABLE TERMS 
FOR A DRUG ABUSE PREVENTION PROGRAM FOR RUNAWAY AND HOMELESS YOUTH 


Component 

What you will do 

Who will do it 

Who you will reach and how 
many 

How long 

Shelter-based services 

Implement 8 drug education 
class each year 

Implement 8 ROPES 
programs each year 

Take 8 field trips each year 
Provide transportation for 
youth not residing in the 
shelter 

2 shelter counselors, 1 
counselor trained in the 

ROPES program 

Approximately. 5 youth 

residing in the shelter for a 
period of two weeks for 
each activity 

Approximately 10 youth 
seeking crisis intervention 
services from the shelter 
for each activity 

Drug abuse education class 
will meet daily for one 
week 

ROPES course will meet daily 
for one week 

Field trip will occur once for 
each class 

Street outreach 

Implement a drop-in center for 
street youth in an area 
where runaway youth 
congregate 

Send out a street van to 
provide food, clothes, and 
drug-related information 

2 counselors (1 man and 1 
woman) who are trained in 
hug prevention counseling 
will staff the drop-in center 

2 street outreach workers - 
one man and one woman 

Approx. 50 runaway youth 
who are on the streets will be 
served each year 

The drop-in center will be 
open from 2:00 - 
10:00 pm every day. 

The street van will go out two 
times per week from 

10:00 pm to 2:00 am. 

Home-based services 

Provide informal family and 
couples counseling in the 
home 2 times per week 
Provide on-call 24 -hour per 
day case management for 
crisis resolution and 
accessing needed services 

2 family therapists and 2 
aides 

Approximately 50 families 
each year of youth who have 
run away or are at risk for 
running away 

Each family will receive 
services for a 3-month 
period 

Follow-up services will be 
provided for 4 weeks 






SAMPLE PARTICIPANT OUTCOME OBJECTIVES STATED IN MEASURABLE TERMS 
FOR A COMPREHENSIVE CHILD DEVELOPMENT PROGRAM 


Components 

Target 

Population 

General Expected 
Outcomes 

Measurable Expected Outcomes 

Home-based 

services 

Family 

Improve family's 
access to 
services 

Improve family 
functioning 

After receiving services in comparison to before receiving services, families will demonstrate: 

• Increases in their knowledge about available services in the community 

• Increases in the number of services they are linked to 

• Decreases in scores on an inventory measuring family stress 

• Increases in scores on an inventory measuring life skills 

Child 

development 

Children 

Improve child 
development 

After receiving services in comparison to before receiving services, children will demonstrate increases in 
scores on a child development inventory 

Children receiving component services will show greater increases in scores than a matched group of 
children who did not receive component services 

Mental health 
services 

Families 

Improve family 
functioning 

After receiving services, in comparison to before receiving services, parents will show: 

• Improvements in their ability to cope with stress 

• Improvements in self-esteem 

Parent 

education 

Parents 

Improve parenting 
practices 

Improve children's 
health 

After receiving services, in comparison to before receiving services, parents will demonstrate: 

• Increases in their knowledge about appropriate parenting practices 

• Increases in their knowledge about nutrition 

• Increases in the number of the nutritious foods they buy and nutritious meals they prepare 

• Increases in the number of well-child doctor's visits they take their children to 

Interagency 

partnerships 

Community 
agencies and 
organizations 

Improve families' 
access to services 

After participating in this component in comparison to before participating in this component, interagency 
collaboration members will demonstrate: 

• Increases in the number of referrals they make to the program 

• Increases in the number of referrals they take from the program 

• Increases in their willingness to change procedures to make access to services easier 




WORKSHEET: PARTICIPANT OUTCOME OBJECTIVES STATED IN MEASURABLE TERMS 





IN THIS CHAPTER... 

Parts of an evaluation plan 

Procedures for managing and 
monitoring an evaluation 
Sample outline 



What Should You 
Include in an 
Evaluation Plan? 

I f you decided to build a house, you probably would hire an 
architect to design the house and draw up the plans. Although 
it is possible to build a house without hiring an architect, this 
professional knows what is and is not structurally possible and 
understands the complex issues relevant to setting the foundation 
and placing the pipes, ducts, and electrical wires. An architect also 
knows what materials to use in various parts of the house and the 
types of materials that are best. However, an architect cannot 
design the house for you unless you tell him or her what you want. 

An evaluation plan is a lot like an architect's plans for a house. It is 
a written document that specifies the evaluation design and details 
the practices and procedures to use to conduct the evaluation. Just 
as you would have an architect develop the plans for your house, it 
is a good idea to have an experienced evaluator develop the plans 
for your evaluation. Similarly, just as an architect cannot design 
your house without input from you, an experienced evaluator 
cannot develop an effective evaluation plan without assistance 
from you and your staff. The evaluator has the technical expertise, 
but you and your staff have the program expertise. Both are 
necessary for a useful evaluation plan. 

If you plan to hire an outside evaluator to head your evaluation 
team, you may want to specify developing the evaluation plan as 
one of the evaluator's responsibilities, with assistance from you and 
program staff. If you plan to conduct an in-house evaluation and 
do not have someone on your evaluation team who is an 
experienced evaluator, this is a critical point at which to seek 
assistance from an evaluation consultant. The consultant can help 



WHAT SHOULD YOU INCLUDE IN AN EVALUATION PLAN? 


You need to prepare an 
evaluation plan well in advance of 
beginning your evaluation, 
especially if a human subjects 
review or Institutional Review 
Board (IRB) review is required by 
your agency when conducting 
research and evaluation activities. 


you prepare the evaluation plan to ensure that your design and 
methodology are technically correct and appropriate for answering 
the evaluation questions. 

This chapter provides information about the necessary ingredients 
to include in an evaluation plan. This information will help you: 

■ Work with an experienced evaluator (either an outside 
evaluator or someone within your agency) to develop the plan. 

■ Review the plan that an outside evaluator has developed to 
make sure all the ingredients are included. 

■ Understand the kinds of things that are required in an 
evaluation and why your outside evaluator or evaluation 
consultant has chosen a specific design or methodology. 

An evaluation plan should be developed at least 2 to 3 months 
before the time you expect to begin the evaluation so that you 
have ample time to have the plan reviewed, make any necessary 
changes, and test out information collection procedures and 
instruments before collecting data. 

Do not begin collecting evaluation information until the plan is 
completed and the instruments have been pilot-tested. A sample 
evaluation plan outline that may be used as a guide appears at the 
end of this chapter. The major sections of the outline are discussed 
below. 

SECTION I. THE EVALUATION FRAMEWORK 

This section can be used to present the program model (discussed 
in Chapter 5), program objectives, evaluation questions, and the 
timeframe for the evaluation (when collection of evaluation 
information will begin and end). It also should include a discussion 
of the context for the evaluation, particularly the aspects of the 
agency, program staff, and participants that may affect the 
evaluation (also discussed in Chapter 5). If an outside evaluator is 
preparing the plan, the evaluator will need your help to prepare 
this section. 

SECTION II. EVALUATING IMPLEMENTATION 
OBJECTIVES - PROCEDURES AND METHODS 

This section should provide detailed descriptions of the practices 
and procedures that will be used to answer evaluation questions 
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NOTE: Data do not have to be 
numerical. Evaluation data can be 
any information collected about 
your program or participants. 


WHEN YOU MAY NEED 
ASSISTANCE FROM AN 
EXPERIENCED 
EVALUATOR 

During the process of developing 
an evaluation plan, you will want 
the assistance from an 
experienced evaluator when you 
are: 

■ Deciding on an evaluation 
design 

■ Selecting or developing data 
collection instruments 

■ Developing the analysis plan 

■ Determining the sampling 

plan (if relevant) 


pertaining to your program's implementation objectives. (Are 
implementation objectives being attained and, if not, why not? 
What barriers were encountered? What has facilitated attainment 
of objectives?) 

Types of information needed. In an evaluation, information is 
often referred to as data. Many people think that the term "data" 
refers to numerical information. In fact, data can be facts, statistics, 
or any other items of information. Therefore, any information that 
is collected about your program or participants can be considered 
evaluation data. 

The types of information needed will be guided by the objective 
you assess. For example, when the objective refers to what you 
plan to do, you must collect information on the types of services, 
activities, or educational/ training products that are developed and 
implemented; who received them; and their duration and intensity. 

When the objective pertains to who will do it, you must collect 
information on the characteristics of program staff (including their 
background and experience), how they were recruited and hired, 
their job descriptions, the training they received to perform their 
jobs, and the general staffing and supervisory arrangements for the 
program. 

When the objective concerns who will participate, you must collect 
information about the characteristics of the participants, the 
numbers of participants, how they were recruited, barriers 
encountered in the recruitment process, and factors that facilitated 
recmitment. 

Sources of necessary information. This refers to where, or from 
whom, you will obtain evaluation information. Again, the selection 
of sources will be guided by the objective you are assessing. For 
example: 

■ Information on services can come from program records or 
from interviews with program staff. 

■ Information on staff can come from program records, 
interviews with agency administrators, staff themselves, and 
program managers. 

■ Information on participants and recruitment strategies can 
come from program records and interviews with program staff 
and administrators. 
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SAMPLING is the process when 
only certain participants, staff, or 
records are used for your 
evaluation. 


■ Information about barriers and facilitators to implementing the 
program can come from interviews with relevant program 
personnel. 

This section of the plan also should include a discussion of how 
confidentiality of information will be maintained. You will need to 
develop participant consent forms that include a description of the 
evaluation objectives and how the information will be used. A 
sample participant consent form is provided at the end of this 
chapter. 

How sources of information will be selected. If your program 
has a large number of staff members or participants, the time and 
cost of the evaluation can be reduced by including only a sample of 
these staff or participants as sources for evaluation information. If 
you decide to sample, you will need the assistance of an 
experienced evaluator to ensure that the sampling procedures 
result in a group of participants or staff that are appropriate for 
your evaluation objectives. Sampling is a complicated process, and 
if you do not sample correctly you run the risk of not being able to 
generalize your evaluation results to your participant population as 
a whole. 

There are a variety of methods for sampling your sources. 

■ You can sample by identifying a specific timeframe for 
collecting evaluation-related information and including only 
those participants who were served during that timeframe. 

■ You can sample by randomly selecting the participants (or 
staff) to be used in the evaluation. For example, you might 
assign case numbers to participants and include only the even- 
numbered cases in your evaluation. 

■ You can sample based on specific criteria, such as length of 
time with the program (for staff) or characteristics of 
participants. 

Methods for collecting information. For each implementation 
objective you are assessing, the evaluation plan must specify how 
information will be collected (the instruments and procedures) and 
who will collect it. To the extent possible, collection of evaluation 
information should be integrated into program operations. For 
example, in direct services programs, the program's intake, 
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For each of your program’s 
objectives, the evaluation plan 
must describe: 

Types of information needed 
Sources of information 
^ How sources will be selected 

Methods for collecting 
information (instruments and 
procedures) 

S Time-frame for collecting 
information 

S Methods for analyzing 
information 


assessment, and termination forms could be designed so that they 
are useful for evaluation purposes as well as for program purposes. 

In training programs, the registration forms for participants can be 
used to collect evaluation-related information as well as provide 
information relevant to conducting the training. If your program 
uses a management information system (MIS) to track services and 
participants, it is possible that it will incorporate much of the 
information that you need for your evaluation. 

There are a number of methods for collecting information 
including structured and open-ended interviews, paper and pencil 
inventories or questionnaires, observations, and systematic reviews 
of program or agency records or documents. The methods you 
select will depend upon the following: 

■ The evidence you need to establish that your objectives were 
attained 

■ Your sources 

■ Your available resources 

Chapter 7 provides more information on these methods. The 
instruments or forms that you will use to collect evaluation 
information should be developed or selected as part of the 
evaluation plan. Do not begin an evaluation until all of the data 
collection instruments are selected or developed. Again, instrument 
development or selection can be a complex process and your 
evaluation team may need assistance from an experienced 
evaluator for this task. 

Confidentiality. An important part of implementing an evaluation 
is ensuring that your participants are aware of what you are doing 
and that they are cooperating with the evaluation voluntarily. 
People should be allowed their privacy, and this means they have 
the right to refuse to give any personal or family information, the 
right to refuse to answer any questions, and even the right to 
refuse to be a part of the evaluation at all. 
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WHY INFORMED CONSENT 
IS SO IMPORTANT. People 
should be allowed their privacy, 
and this means that they have the 
right to refuse to give any 
personal or family information, 
the right to refuse to answer any 
questions in the evaluation, and 
even the right to refuse to be a 
part of the evaluation at all. For 
participant consent to be 
informed, you must clearly 
explain these rights, as well as any 
risks associated with the 
evaluation. 


NOTE: To determine whether 
your participants changed, you 
need to collect information about 
them before they begin (and 
before they receive services or 
training) and after they complete 
your program. 


Explain the evaluation activities and what will be required of them 
as part of the evaluation effort. Tell them that their name will not 
be used and that the information they provide will not be linked to 
them. Then, have them sign an informed consent form that 
documents that they understand the scope of the evaluation, know 
what is expected of them, agree (or disagree) to participate, and 
understand they have the right to refuse to give any information. 
They should also understand that they may drop out of the 
evaluation at any time without losing any program services. If 
children are involved, you must get the permission of their parents 
or guardians concerning their participation in the evaluation. 

A sample informed consent form appears at the end of this 
chapter. Sometimes programs will have participants complete this 
form at the same time that they complete forms agreeing to 
participate in the program, or agreeing to let their children 
participate. This reduces the time needed for the evaluator to 
secure informed consent. 

Timeframe for collecting information. Although you will have 
already specified a general timeframe for the evaluation, you will 
need to specify a time frame for collecting data relevant to each 
implementation objective. Times for data collection will again be 
guided by the objective under assessment. You should be sure to 
consider collecting evaluation at the same time for all participants; 
for example, after they have been in the program for 6 months. 

Methods for analyzing information. This section of an 
evaluation plan describes the practices and procedures for use in 
analyzing the evaluation information. For assessing program 
implementation, the analyses will be primarily descriptive and may 
involve tabulating frequencies (of services and participant 
characteristics) and classifying narrative information into 
meaningful categories, such as types of barriers encountered, 
strategies for overcoming barriers, and types of facilitating factors. 
An experienced evaluator can help your evaluation team design an 
analysis plan that will maximize the benefits of the evaluation for 
the program and for program staff. More information on analyzing 
program implementation information is provided in Chapter 8. 

SECTION III. EVALUATING PARTICIPANT 
OUTCOME OBJECTIVES 

The practices and procedures for evaluating attainment of 
participant outcome objectives are similar to those for evaluating 
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implementation objectives. However, this part of your evaluation 
plan will need to address a few additional issues. 

Selecting your evaluation design. A plan for evaluating 
participant outcome objectives must include a description of the 
evaluation design. Again, the assistance of an experienced 
evaluator (either an outside evaluator, consultant, or someone 
within your agency) is critical at this juncture. 

The evaluation design must allow you to answer these basic 
questions about your participants: 

■ Did program participants demonstrate changes in knowledge, attitudes, 
behaviors, or awareness ? 


■ Were the changes the result of the program's interventions ? 

Answering the first question concerns assessment of participant 
outcomes; answering the second concerns assessment of the 
impact of the program on the participants. 

Two commonly used evaluation designs are: 

■ Pre-intervention and post-intervention assessments 

■ Pre-intervention and post-intervention assessments using a comparison or 



Using a control or comparison 
group is the best way to 
determine the impact of your 
program on participants. 

A control group is formed by 
randomly assigning participants 
to either a treatment group 
(which receives services) or a 
control group (which receives no 
services). 

A comparison group is used 
when you cannot randomly assign 
participants; instead you identify a 
group of people similar to your 
participants but who do not 
receive your program’s services. 


A pre- and post-intervention design involves collecting 
information only on program participants. This information is 
collected at least twice: once before participants begin the program 
and again either immediately or some time after they complete or 
leave the program. You can collect outcome information as often 
as you like after participants enter the program, but you must 
collect information on participants before they enter the program. 
This is called baseline information and is essential for demonstrating 
that a change occurred. 

If you are implementing an education or training program, this 
type of design can be effective for evaluating immediate changes in 
participants' knowledge and attitudes. In these types of programs, 
you can assess participants' knowledge and attitudes prior to the 
training and immediately after training with some degree of 
certainty that any observed changes resulted from your 
interventions. 
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NOTE: Random assignment 
does not mean that control group 
members cannot receive any 
services or training. They may 
remain on a waiting list and 
participate in your program after 
the evaluation information has 
been collected. 


However, if you want to assess longer-term outcomes of 
training and education programs or any outcomes of service 
delivery programs, the pre-intervention and post-intervention 
design by itself is not recommended. Collecting information 
only on program participants does not allow you to answer the 
question: Were participant changes the result of program 
interventions? The changes may have occurred as a result of other 
interventions, or are changes that might have occurred without any 
intervention at all. 

To be able to attribute participant changes to your program's 
intervention, you need to use a pre- and post-intervention design 
that incorporates a comparison or control group. In this design, 
two groups of individuals are included in your evaluation. 

■ The treatment group (individuals who participate in your 
program). 

■ The non treatment group (individuals who are similar to those 
in the treatment group, but who do not receive the same 
services as the treatment group). 

The non treatment group is called a control group if all eligible 
program participants are randomly assigned to the treatment and 
non treatment groups. Random assignment means that members 
of both groups can be assumed to be similar with respect to all key 
characteristics except program participation. Thus, potential 
sources of biases are "controlled." A comparison group is a non 
treatment group where you do not randomly assign people. A 
comparison group could be families from another program, 
children from another school, or former program participants. 

Evaluation designs that include a control group are the most 
rigorous. Because the key feature of these designs is random 
assignment, they are also known as randomized experiments or 
randomized controlled trials. They provide the strongest evidence 
that changes in participant outcomes are due to a program 
intervention. 

Although using a control group greatly strengthens your 
evaluation, there are barriers to implementing this design option. 
Program staff may view random assignment as unethical because it 
deprives eligible participants of needed services. As a result, staff 
sometimes will prioritize eligible participants rather than use 
random assignment, or staff may simply refuse to assign 
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individuals to the control group. Staff from other agencies may 
also feel random assignment is unethical and may refuse to refer 
individuals to your program. 

To avoid these potential barriers, educate staff from your program 
and from other agencies in your community about the benefits of 
the random assignment process. No one would argue with the 
belief that it is important to provide services to individuals who 
need them. However, it is also important to find out if those 
services actually work. The random assignment process helps you 
determine whether or not your program's services are having the 
anticipated effect on participants. Staff from your program and 
from other agencies also must be informed that random 
assignment does not mean that control group members cannot 
receive any services or training. They may participate in the 
program after the evaluation data have been collected, or they may 
receive other types of services or training. 

Another potential barrier to using a control group is the number of 
program participant that are recruited. If you find that you are 
recmiting fewer participants than you originally anticipated, you 
may not want to randomly assign participants to a control group 
because it would reduce the size of your service population. 

A final barrier is the difficulty of enlisting control group members 
in the evaluation process. Because control group members have 
not participated in the program, they are unlikely to have an 
interest in the evaluation and may refuse to be interviewed or 
complete a questionnaire. Some evaluation efforts set aside funds 
to provide money or other incentives to encourage both control 
group and treatment group members to participate in the 
evaluation. Although there is some potential for bias in this 
situation, it is usually outweighed by the need to collect 
information from control group members. 

If you are implementing a program in which random assignment 
of participants to treatment and control groups is not possible, you 
will need to identify a group of individuals or families who are 
similar to those participating in your program whom you can 
assess as part of your evaluation. This group is called a comparison 
group, and these designs are called quasi-experiments. Similar to a 
control group, members of a comparison group may receive other 
types of services or no services at all. Although using comparison 
groups means that programs do not have to deny services to 
eligible participants, you cannot be sure that the two groups are 
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REMEMBER: You should notify 
your funder whenever you are 
planning major changes to your 
project or evaluation. 


completely similar, which is why it is more difficult to attribute 
changes in participant outcomes to a program intervention. You 
should consult with your outside evaluator or evaluation consultant 
regarding the statistical techniques and design variations that can 
strengthen quasi-experimental designs. For example, you may have 
to collect enough information at baseline to try and statistically 
control for potential differences as part of your analyses. 

Comparison group members may be participants in other 
programs provided by your agency or in programs offered by other 
agencies. If you plan to use a comparison group, you must make 
sure that this group will be available for assessments during the 
time frame of your evaluation. Also, be aware that comparison 
group members, like control group members, are difficult to enlist 
in an evaluation. The evaluation plan will need to specify strategies 
for encouraging non treatment group members to take part in the 
evaluation. 

Pilot-testing information collection instruments. Your plans 
for evaluating participant outcome objectives will need to include a 
discussion of plans for pilot-testing and revising information 
collection instmments. Chapter 7 provides information on pilot- 
testing instruments. 

Analyzing participant outcome information. The plan for 
evaluating participant outcomes must include a comprehensive 
data analysis plan. The analyses must be structured to answer the 
questions about whether change occurred and whether these 
changes can be attributed to the program. A more detailed 
discussion on analyzing information on participant outcomes is 
provided in Chapter 8. 

SECTION IV. PROCEDURES FOR MANAGING AND 
MONITORING THE EVALUATION 

This section of the evaluation plan can be used to describe the 
practices and procedures you expect to use to manage the 
evaluation. If staff are to be responsible for data collection, you 
will need to describe how they will be trained and monitored. You 
may want to develop a data collection manual that staff can use. 
This will ensure consistency in information collection and will be 
useful for staff who are hired after the evaluation begins. Chapter 7 
discusses various types of evaluation monitoring activities. 

This final section of the evaluation plan also should include a 
discussion of how changes in program operations will be handled 
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in the evaluation. For example, if a particular service or program 
component is discontinued or added to the program, you will need 
to have procedures for documenting the time that this change 
occurred, the reasons for the change, and whether particular 
participants were involved in the program prior to or after the 
change. This will help determine whether the change had any 
impact on attainment of expected outcomes. 

Once you and your experienced evaluator have completed the 
evaluation plan, it is a good idea to have it reviewed by selected 
individuals for their comments and suggestions. Potential 
reviewers include the following: 

■ Agency administrators who can determine whether the 
evaluation plan is consistent with the agency's resources and 
evaluation objectives. 

■ Program staff who can provide feedback on whether the 
evaluation will involve an excessive burden for them and 
whether it is appropriate for program participants. 

■ Advisory board members who can assess whether the 
evaluation will provide the type of information most important 
to know. 

■ Participants and community members who can determine if 
the evaluation instmments and procedures are culturally 
sensitive and appropriate. 

After the evaluation plan is complete and the instruments pilot 
tested, you are ready to begin collecting evaluation information. 
Because this process is so critical to the success of an evaluation, 
the major issues pertaining to information collection discussed in 
more detail in the following chapter. 
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SAMPLE OUTLINE FOR EVALUATION PLAN 


I. Evaluation framework 

A. What you are going to evaluate 

1. Program model (assumptions about target population, interventions, immediate outcomes, 
intermediate outcomes, and final outcomes) 

2. Program implementation objectives (stated in general and then measurable terms) 

a. What you plan to do and how 

b. Who will do it 

c. Participant population and recruitment strategies 

3. Participant outcome objectives (stated in general and then measurable terms) 

4. Context for the evaluation 

B. Questions to be addressed in the evaluation 

1. Are implementation objectives being attained? If not, why (that is, what barriers or problems 
have been encountered)? What kinds of things facilitated implementation? 

2. Are participant outcome objectives being attained? If not, why (that is, what barriers or 
problems have been encountered)? What kinds of things facilitated attainment of participant 
outcomes? 

a. Do participant outcomes vary as a function of program features? (That is, which 
aspects of the program are most predictive of expected outcomes?) 

b. Do participant outcomes vary as a function of characteristics of the participants or 
staff? 

C. Timeframe for the evaluation 

1. When data collection will begin and end 

2. How and why timeframe was selected 
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II. Evaluating implementation objectives - procedures and methods 


(question 1: Are implementation objectives being attained, and if not, why not?) 


A. Objective 1 (state objective in measurable terms) 


1. Type of information needed to determine if objective 1 is being attained and to assess barriers 
and facilitators 


2. Sources of information (that is, where you plan to get the information including staff, 
participants, program documents). Be sure to include your plans for maintaining confidentiality 
of the information obtained during the evaluation 

3. How sources of information were selected 


4. Time frame for collecting information 

5. Methods for collecting the information (such as interviews, paper and pencil instruments, 
observations, records reviews) 

6. Methods for analyzing the information to determine whether the objective was attained (that 
is, tabulation of frequencies, assessment of relationships between or among variables) 


B. Repeat this information for each implementation objective being assessed in the evaluation 


III. Evaluating participant outcome objectives-procedures and methods 

(question 2: Are participant outcome objectives being attained and if not, why not?) 


A. Evaluation design 

B. Objective 1 (state outcome objective in measurable terms) 


1. Types of information needed to determine if objective 1 is being attained (that is, what 
evidence will you use to demonstrate the change?) 

2. Methods of collecting that information (for example, questionnaires, observations, surveys, 
interviews) and plans for pilot-testing information collection methods 

3. Sources of information (such as program staff, participants, agency staff, program managers, 
etc.) and sampling plan, if relevant 

4. Timeframe for collecting information 

5. Methods for analyzing the information to determine whether the objective was attained (i.e., 
tabulation of frequencies, assessment of relationships between or among variables using 
statistical tests) 


C. Repeat this information for each participant outcome objective being assessed in the evaluation 
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IV. Procedures for managing and monitoring the evaluation 

A. Procedures for training staff to collect evaluation-related information 

B. Procedures for conducting quality control checks of the information collection process 

C. Timelines for collecting, analyzing, and reporting information, including procedures for providing 
evaluation-related feedback to program managers and staff 
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SAMPLE INFORMED CONSENT FORM 

We would like you to participate in the Evaluation of [program name]. Your participation is important to us and will help 
us assess the effectiveness of the program. As a participant in [program name] we will ask you to [complete a 
questionnaire, answer questions in an interview, or other task]. 

We will keep all of your answers confidential. Your name will never be included in any reports and none of your answers 
will be linked to you in any way. The information that you provide will be combined with information from everyone else 
participating in the study. 

[If information/data collection includes questions relevant to behaviors such as child abuse, drug abuse, or suicidal 
behaviors, the program should make clear its potential legal obligation to report this information - and that confidentiality 
may be broken in these cases. Make sure that you know what your legal reporting requirements are before you begin 
your evaluation.] 

You do not have to participate in the evaluation. Even if you agree to participate now, you may stop participating at any 
time or refuse to answer any question. Refusing to be part of the evaluation will not affect your participation or the 
services you receive in [program name]. 

If you have any questions about the study you may call [name and telephone number of evaluator, program manager or 
community advocate ]. 

By signing below, you confirm that this form has been explained to you and that you understand it. 

Please Check One: 

□ AGREE TO PARTICIPATE 

□ DO NOT AGREE TO PARTICIPATE 

Signed: 

Participant or Parent/Guardian 


Date: 
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IN THIS CHAPTER... 
Determining information needs 
and sources 

Identifying effective instruments 

Developing data collection 
procedures 

Pilot testing and monitoring 
activities 



How Do You Get 
the Information 
You Need for 
Your Evaluation? 

A s Chapter 6 noted, a major section of your evaluation plan 
concerns evaluation information - what kinds of 
information you need, what the sources for this 
information will be, and what procedures you use to 
collect it. Because these issues are so critical to the success of your 
evaluation effort, they are discussed in more detail in this chapter. 

In a program evaluation, the information you collect is similar to 
the materials you use when you build a house. If you were to build 
a house, you would be very concerned about the quality of the 
materials used. High-quality materials ensure a strong and durable 
house. In an evaluation, the quality of the information you collect 
also affects its strength and durability. The higher the quality of the 
information collected, the better the evaluation. 

At the end of the chapter, there are two worksheets to help you 
plan out the data collection process. One is a sample worksheet 
completed for a drug abuse prevention program for mnaway and 
homeless youth, and the other is a blank worksheet that you and 
your evaluation team can complete together. The following 
sections cover each column of the worksheet. 

WHAT SPECIFIC INFORMATION DO YOU NEED TO 
ADDRESS OBJECTIVES? 

Using the worksheet, fill in your program implementation (or 
participant outcome objectives) in column 1. Make sure that these 
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When making decisions about 
your information needs, you and 
other members of your 
evaluation team may decide that 
certain types of information 
would be interesting to collect. 
However, if the information does 
not related directly to your 
implementation or outcome 
objectives, you should resist this 
urge. By staying focused on your 
specific information needs, you 
will be able to keep the time and 
costs of the evaluation to a 
minimum. 


objectives are stated in measurable terms. Your statement of objectives 
in measurable terms will determine the kinds of information you need 
and will avoid the problem of collecting more information than is 
actually necessary. 

Next, complete column 2 by specifying the information that addresses 
each objective. This information is sometimes referred to as the data 
elements. For example, if two of your measurable participant outcome 
objectives are to improve youth's grades and scores on academic tests 
and reduce their incidence of behavioral problems as reported by 
teachers and student self-reports, you will need to collect the following 
information: 

■ Student grades 

■ Academic test scores 

■ Number of behavior or discipline reports 

■ Teacher assessments of classroom behaviors 

■ Student self-assessments of classroom behaviors 
These items are the data elements. 

WHAT ARE THE BEST SOURCES? 

Column 3 can be used to identify appropriate sources for specific 
evaluation data. For every data element, there may be a range of 
potential sources, including: 

■ Program records (case records, registration records, academic 
records, and other information) 

■ Program management information systems 

■ Program reports and documents 

■ Program staff 

■ Program participants 

■ Family members of participants 

■ Members of a control or comparison group 
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HOW DO YOU GET THE INFORMATION YOU NEED? 


OBSERVATION of participant 
behavior (especially from 
children) can be a very good way 
to collect evaluation information. 
However, careful training of 
observers is necessary, in addition 
to well-designed and validated 
observation checklists. 


■ Staff of collaborating agencies 

■ Records from other agencies (such as health agencies, schools, 
criminal justice agencies, mental health agencies, child welfare 
agencies, or direct service agencies) 

■ Community leaders 

■ Outside experts 

■ The general public 

■ National databases 

In deciding the best sources for information, your evaluation team 
will need to answer three questions: 

S What source is likely to provide the most accurate information? 

S What source is the least costly or time consuming? 

S Will collecting information from a particular source pose an excessive 
burden on that person? 

The judgment regarding accuracy is the most important decision. 
For example, it may be less costly or time consuming to obtain 
information about services from interviews with program staff, but 
staff may not provide as accurate information about services as 
may be obtained from case records or program logs. 

When you interview staff, you are relying on their memories of 
what happened, but when you review case records or logs, you 
should be able to get information about what actually did happen. 
If you choose to use case records or program logs to obtain 
evaluation-relevant data, you will need to make sure that staff are 
consistent in recording evaluation information in the records. 
Sometimes case record reviews can be difficult to use for 
evaluation purposes because they are incomplete or do not report 
information in a consistent manner. 

Another strategy is to identify existing information on your 
participants. Although your program may not collect certain 
information, other programs and agencies may. You might want to 
seek the cooperation of other agencies to obtain their data, or 
develop a collaboration that supports your evaluation. 
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WHAT ARE THE MOST EFFECTIVE DATA 
COLLECTION INSTRUMENTS? 

Column 4 identifies the instruments that you will use to collect the 
data from specified sources. Some options for information 
collection instruments include the following: 


IN DECIDING THE BEST 
SOURCES FOR 
INFORMATION, CONSIDER 

S The most accurate source 
S The burden on the source 

Y The availability and 
accessibility of the source 
'C Existing sources 


The types of instruments selected should be guided by your data 
elements. For example, information on barriers or facilitators to 
program implementation would be best obtained through oral 
interviews with program administrators and staff. Information on 
services provided may be more accurate if obtained by using a case 
record or program log extraction form. 

Information on family functioning may be best obtained through 
observations or questionnaires designed to assess particular aspects 
of family relationships and behaviors. Focus group interviews are 
not always useful for collecting information on individual 
participant outcomes, but may be used effectively to assess 
participants' perceptions of a program. 

Instruments for evaluating program implementation 
objectives. Your evaluation team will probably need to develop 
instruments to collect information on program implementation 
objectives. This is not a complicated process. You must pay 
attention to your information needs and potential sources and 
develop instmments designed specifically to obtain that 
information from that source. For example, if you want to collect 
information on planned services and activities from program 
planners, it is possible to construct an interview instmment that 
includes the following questions: 

V Why was the decision made to develop ( the particular service or activity j? 

Y Who was involved in making this decision? 


Written surveys or questionnaires 

Oral interviews (either in person or on the telephone) or focus 
group interviews (either stmctured or unstructured) 

Extraction forms to be used for written records (such as case 
records or existing databases) 

Observation forms or checklists to be used to assess 
participants' or staff members' behaviors 
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OTHER IDEAS. Sometimes 
programs can use nontraditional 
and creative methods of 
documenting program operations 
and participant successes. These 
methods generally are used to 
supplement the evaluation 
activities already under way. 

Some methods you can consider 
to supplement the information 
collected as part of your 
evaluation include the following: 

■ Audio and video recordings 

■ Participant journals 

■ Participant plays and skits 

■ Storybooks 


d What plans were made to ensure the cultural relevant y of (the particular 
service or activity j? 

If case records or logs are viewed as appropriate sources for 
evaluation information, you will need to develop a case record or 
program log extraction form. For example, if you want to collect 
information on actual services or activities, you may design a 
records extraction form that includes the following items: 

d How many times was ( the particular activity or service) provided to each 
participant ? 

d Who provided or implemented ( the particular activity or service )? 

d What was the intensity of (the particular activity or service )? (How long 
was it provided for each participant at each time)? 

d What was the duration of (the particular activity or service )? (What was 
the timeframe during which the participant received or participated in the 
activity or service ?) 

Instruments for evaluating participant outcome objectives. 

Participant outcome objectives can be assessed using a variety of 
instruments, depending on your information needs. If your 
evaluation team decides to use interview instruments, observations, 
or existing records to collect participant outcome information, you 
will probably need to develop these instmments. In these 
situations, you would follow the same guidelines as you would use 
to develop instmments to assess program implementation 
objectives. 

If your evaluation team decides to use questionnaires or 
assessment inventories to collect information on participant 
outcomes, you have the option of selecting existing instmments or 
developing your own. Many existing instmments can be used to 
assess participant outcomes, particularly with respect to child abuse 
potential, substance use, family cohesion, family stress, behavioral 
patterns, and so on. It is not possible to identify specific 
instruments or inventories in this manual as particularly 
noteworthy or useful, because the usefulness of an instrument 
depends to a large extent on the nature of your program and your 
participant outcome objectives. If you do not have someone on 
your evaluation team who is knowledgeable regarding existing 
assessment instmments, this would be a critical time to enlist the 
assistance of an outside consultant to identify appropriate 
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NORMED and NORMAL are 
statistical terms that refer to the 
distribution of different scores in 
a group of people. These terms 
do not imply dysfunction or 
abnormality among participants. 


instruments. Some resources for existing instruments are provided 
in the appendix. 

There are advantages and disadvantages to using existing 
instruments. The primary advantages of using existing 
instruments or inventories are noted on the following page: 

They often, but not always, are standardized. This means that 
the instrument has been administered to a very large population 
and the scores have been "normed" for that population. When an 
instrument has been "normed," it means that a specified range of 
scores is considered "normal," whereas scores in another range are 
considered "non-normal." Non-normal scores on instruments 
assessing child abuse potential, substance use, family cohesion, and 
the like may be indicators of potential problem behaviors. 

They usually, but not always, have been established as valid 
and reliable. An instrument is valid if it measures what it is 
supposed to measure. It is reliable if individuals' responses to the 
instrument are consistent over time or within the instrument. 

The primary disadvantages of using existing instmments are as 
follows: 

They are not always appropriate for all cultural or ethnic 
populations. Scores that are "normed" on one cultural group may 
not reflect the norm of members of another cultural group. 
Translating the instrument into another language is not sufficient 
to make it culturally appropriate. The items and scoring system 
must reflect the norms, values, and traditions of the given cultural 
group. 

They may not be useful for your program. Your participant 
outcome objectives and the interventions you developed to attain 
those objectives may not match what is being assessed by a 
standardized instmment. For example, if you want to evaluate the 
effects that a tutoring program has on runaway and homeless 
youth, an instrument measuring depression may not be useful. 

If an outside consultant selects an instmment for your program 
evaluation, make sure that you and other members of the 
evaluation team review each item on the instmment to ensure that 
the information it asks for is consistent with your expectations 
about how program participants will change. 
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CULTURAL RELEVANCE. 
Your instruments should be 
sensitive to issues and concerns 
of your participant group. You 
will want to know: 

^ Do participants understand the 
terms the instruments use? Is the 
language at a level everyone can 
understand? 

^ Are concepts dealt with in the 
instruments familiar to 
participants? 

^ Do the instruments deal with 
concepts in a thoughtful and 
nonintrusive manner? 

^ Do the questions support the 
values of the participant group? 


If your evaluation team is unable to find an appropriate existing 
instrument to assess participant outcome objectives, they will need 
to develop one. Again, if there is no one on your team who has 
expertise in developing assessment instmments, you will need the 
assistance of an outside consultant for this task. 

Whether you decide to use an existing instrument or develop one, 
the instmment used should meet the following criteria: 

Y It should measure a domain addressed by your program. 

If you are providing parenting training, you would want an 
instrument to measure changes in parenting knowledge, skills, and 
behaviors, not an instrument measuring self-esteem, substance use, 
or personality type. 

Y It should be appropriate for your participants in terms of 
age or developmental level, language, and ease of use. These 
characteristics can be checked by conducting focus groups of 
participants or pilot testing the instmments. 

Y It should respect and reflect the participants' cultural 
backgrounds. The definitions, concepts, and items in the 
instrument should be relevant to the participants' community and 
experience. 

Y The respondent should be able to complete the 
instrument in a reasonable timeframe. Again, careful pilot 
testing can uncover any difficulties. 

WHAT PROCEDURES SHOULD YOU USE TO 
COLLECT DATA? 

It is critical that the evaluation team establish a set of procedures 
to ensure that the information will be collected in a consistent and 
systematic manner. Information collection procedures should 
include: 

When the information will be collected. This will depend on the 
schedule the evaluation team has established for the specific time 
intervals that information must be collected. 

Where the information will be collected. This is particularly 
relevant when information is to be collected from program 
participants. The evaluation team must decide whether the 
information will be collected in the program facility, in the 
participants' homes, or in some other location. It is a good idea to 
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PILOT TESTING of data 
collection instruments and 
procedures allows you to identify 
and correct problem questions 
and procedures before you begin 
your evaluation. 


be consistent about where you collect information. For example, 
participants may provide different responses in their own home 
environments than they would in an agency office setting. 

Who will collect the information. In some situations, you will 
need to be sure that information collectors meet certain criteria. 
For example, they may need to be familiar with the culture or the 
language of the individuals they are interviewing or observing. 
Administering some instmments also may require that the 
information collector has experience with the instruments or has 
clinical experience or training. 

How the information will be collected. This refers to 
procedures for administering the instmments. Will they be 
administered as a group or individually? If you are collecting 
information from children, will other family members be present? 
If you are collecting information from individuals with a low level 
of literacy, will the data collectors read the items to them? The 
methods you use will depend in large part on the type of program 
and the characteristics of the participants. Training and education 
programs, for example, may have participants complete 
instruments in a group setting. Service delivery programs may find 
it more appropriate to individually administer instruments. 

Everyone involved in collecting evaluation information must be 
trained in data collection procedures. Training should include: 

d An item-by-item review of each of the instruments to be used in data 
collection, including a discussion of the meaning of each item, why it was 
included in the instrument, and how it is to be completed 

d A review of all instructions on administering or using the instruments, 
including instructions to the respondents 

d A discussion of potential problems that may arise in administering the 
instrument, including procedures for resolving the problems 

d A practice session during which data collection staff administer the 
instrument to one another, use it to extract information from existing case 
records or program logs, or complete it themselves, if it is a written 
questionnaire 

d A discussion of respondent confidentiality, including administering an 
informed consent form, answering respondents' questions about confidentiality, 
keeping completed instruments in a safe place, and procedures for submitting 
instruments to the appropriate person 


70 



HOW DO YOU GET THE INFORMATION YOU NEED? 


^ A discussion of the need for frequent reviews and checks of the data and 
for meetings of data collectors to ensure data collection continues to be consistent. 

It is useful to develop a manual that describes precisely what is 
expected in the information collection process. This will be a 
handy reference for data collection staff and will be useful for new 
staff who were hired after the initial evaluation training occurred. 

WHAT CAN BE DONE TO ENSURE THE 
EFFECTIVENESS OF INSTRUMENTS AND 
PROCEDURES? 

Even after you have selected or constructed the instruments and 
trained the data-collection staff, you are not yet ready to begin 
collecting data. Before you can actually begin collecting evaluation 
information, you must "pilot test" your instmments and 
procedures. The pilot test will determine whether the instruments 
and procedures are effective - that they obtain the information 
needed for the evaluation, without being excessively burdensome 
to the respondents, and that they are appropriate for the program 
participant population. 

You may pilot test your instmments on a small sample of program 
records or individuals who are similar to your program 
participants. You can use a sample of your own program's 
participants who will not participate in the actual evaluation or a 
group of participants in another similar program offered by your 
agency or by another agency in your community. 

The kinds of information that can be obtained from a pilot test 
include: 

How long it takes to complete interviews, extract information from 
records, or fill out questionnaires 

Whether self administered questionnaires can be completed by participants 
without assistance from staff 

Whether the necessary records are readily available, complete, and consistently 
maintained 

Whether the necessary information can be collected in the established time frame 
Whether instruments and procedures are culturally appropriate 
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Whether the notification procedures ( letters , informed consent, and the like j are 
easily implemented and executed 

To the extent possible, pilot testing should be done by data 
collection staff. Ask them to take notes and make comments on 
the process of administering or using each instrument. Then 
review these notes and comments to determine whether changes 
are needed in the instmments or procedures. As part of pilot 
testing, instruments should be reviewed to assess the number of 
incomplete answers, unlikely answers, comments on items that 
may be included in the margins, or other indicators that revisions 
are necessary. 

In addition, you can ask questions of participants after the pilot 
test to obtain their comments on the instruments and procedures. 
Frequently, after pilot testing the evaluation team will need to 
improve the wording of some questions or instructions to the 
respondent and delete or add items. 

HOW CAN YOU MONITOR DATA COLLECTION 
ACTIVITIES? 

Once data collection begins, this task will require careful 
monitoring to ensure consistency in the process. Nothing is more 
damaging to an evaluation effort than information collection 
instruments that have been incorrectly or inconsistently 
administered, or that are incomplete. 

There are various activities that can be undertaken as part of the 
monitoring process. 

Establish a routine and timeframe for submitting completed 
instruments. This may be included in your data collection manual. 
It is a good idea to have instmments submitted to the appropriate 
member of the evaluation team immediately after completion. That 
person can then review the instmments and make sure that they 
are being completed correctly. This will allow problems to be 
identified and resolved immediately. You may need to retrain some 
members of the staff responsible for data collection or have a 
group meeting to re-emphasize a particular procedure or activity. 

Conduct random observations of the data collection process. 

A member of the evaluation team may be assigned the 
responsibility of observing the data collection process at various 
times during the evaluation. This person, for example, may sit in 
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It is helpful to all staff, especially 
those who join your program 
after an evaluation begins, to 
have a procedures manual that 
describes all the data collection 
instruments and procedures. 


on an interview session to make sure that all of the procedures are 
being correctly conducted. 

Conduct random checks of respondents. As an additional 
quality control measure, someone on the evaluation team may be 
assigned the responsibility of checking with a sample of 
respondents on a routine basis to determine whether the 
instruments were administered in the expected manner. This 
individual may ask respondents if they were given the informed 
consent form to sign and if it was explained to them, where they 
were interviewed, whether their questions about the interview were 
answered, and whether they felt the attitude or demeanor of the 
interviewer was appropriate. 

Keep completed interview forms in a secure place. This will 
ensure that instruments are not lost and that confidentiality is 
maintained. Completed data collection instruments should not be 
left lying around, and access to this information should be limited. 
You may want to consider number-coding the forms rather than 
using names, though keeping a secured data base that connects the 
names to numbers. 

Encourage staff to view the evaluation as an important part of 
the program. If program staff are given the responsibility for data 
collection, they will need support from you for this activity. Their 
first priority usually is providing services or training to participants 
and collecting evaluation information may not be valued. You will 
need to emphasize to your staff that the evaluation is part of the 
program and that evaluation information can help them improve 
their services or training to participants. 

Once evaluation information is collected, you can begin to analyze 
it. To maximize the benefits of the evaluation to you, program 
staff, and program participants, this process should take place on 
an ongoing basis or at specified intervals during the evaluation. 
Information on the procedures for analyzing and interpreting 
evaluation information are discussed in the following chapter. 
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SAMPLE DATA COLLECTION PLAN FOR PROGRAM IMPLEMENTATION OBJECTIVES FOR A DRUG ABUSE PREVENTION PROGRAM 

FOR RUNAWAY AND HOMELESS YOUTH 


Implementation Objectives 

Data Elements 

Data Sources 

Data 

Collection 

Instruments 

When 

Collected 

Implement 8 drug education classes, ROPES programs, 
and field trips each year. Provide transportation for youth 
not residing in the shelter 

Planned services and activities 

Program planners 

Interview 

forms 

Once at 
program onset 


Actual services and activities 

Program managers 
Program staff 

Interview 

forms 

Every 3 
months 


Differences between planned and 
actual services 

Program manager 

Interview 

forms 

Every 3 
months 


Barriers and facilitators 

Program manager 
Program staff 

Interview 

forms 

Every 3 
months 

Staff program with 2 drug education counselors, 1 ROPES 
Program facilitator, and 1 driver. Use 2 volunteers for each 
field trip 

Planned staffing 

Program manager 

Interview 

forms 

Once at 
program onset 


Actual staffing 

Program manager 
Program staff 

Interview 

forms 

Every 3 
months 


Differences between planned and 
actual staffing 

Program manager 
Program staff 

Interview 

forms 

Every 3 
months 


Barriers and facilitators 

Program manager 
Program staff 

Interview 

forms 

Every 3 
months 




Target 5 youth residing in shelter and 10 youth receiving 
crisis intervention services for each service. Target 10 non- 
shelter youth for transportation for each session. 

Planned target population and 
numbers served 

Program planners 

Interview 

forms 

Once at 
program onset 


Actual target population 

Program records 

Records 

extraction 

form 

Every 3 
months 


Differences between planned and 
actual target population 

Program managers 
Program staff 

Interview 

forms 

Every 3 
months 


Barriers and facilitators 

Program managers 
Program staff 

Interview 

forms 

Every 3 
months 

Each session of the drug education classes and ROPES 
programs will be provided for 7 consecutive days. Drug 
education classes will be provided for 2 hours in the a.m., 
and the ROPES course for 4 hours in the p.m. 

Planned duration of services 

Program planners 

Interview 

forms 

Once at 
program onset 


Actual duration of services 

Program manager 
Program records 

Interview 

forms 

Record 

extraction 

forms 

Every 3 
months 


Differences between actual and 
planned duration of services 

Program manager 
Program records 

Interview 

forms 

Record 

extraction 

forms 

Every 3 
months 


Barriers and facilitators 

Program manager 
Program staff 

Interview 

forms 

Every 3 
months 






In this chapter... 
Analyzing information on 
objectives 

Understanding statistical 

procedures 

Using results 



How Do You 
Make Sense of 
Evaluation 
Information? 

F or evaluation information to be useful, it must be analyzed and 
interpreted. Many program managers and staff are intimidated 
by this activity, believing that it is best left to an expert. This is 
only partially true. If your evaluation team does not include 
someone who is experienced in analyzing qualitative and quantitative 
evaluation data, you will need to seek the assistance of an outside 
consultant for this task. However, it is important for you and all other 
members of the evaluation team to participate in the analysis activities. 
This is the only way to ensure that the analyses will answer your 
evaluation questions, not ones that an outside consultant may want to 
answer. 

Think again about building a house. You may look at a set of 
blueprints and see only a lot of lines, numbers, and arrows. But 
when a builder looks at the blueprints, this person sees exactly 
what needs to be done to build the house and understands all of 
the technical requirements. This is why most people hire an expert 
to build one. However, hiring an expert builder does not mean that 
you do not need to participate in the building process. You need to 
make sure that the house the builder is working on is the house 
you want, not one that the builder wants. 

This chapter will not tell you how to analyze evaluation data. 
Instead, it provides some basic information about different 
procedures for analyzing evaluation data to help you understand 
and participate more fully in this process. There are many ways to 



THE PROGRAM MANAGER’S GUIDE TO EVALUATION 


analyze and interpret evaluation information. The methods 
discussed in this chapter are not the only methods one can use. 
Whatever methods the evaluation team decides to use, it is 
important to realize that analysis procedures must be guided by the 
evaluation questions. The following evaluation questions are 
discussed throughout this manual: 

Are program implementation objectives being attained ? If not, why not ? What 
types of things were barriers to or facilitated attaining program implementation 
objectives ? 

Are participant outcome objectives being attained ? If not, why not ? What types 
of things were barriers to or facilitated attaining participant outcome objectives ? 

The following sections discuss procedures for analyzing evaluation 
information to answer both of these questions. 

ANALYZING INFORMATION ABOUT PROGRAM 
IMPLEMENTATION OBJECTIVES 

In this manual, the basic program implementation objectives have 
been described as follows: 

What you plan to do 

Who will do it 

Whom you plan to reach (your expected participant population j and with what 
intensity and duration 

How many you expect to reach 

You can analyze information about attainment of program 
implementation using a descriptive process. You describe what 
you did (or are doing), who did it, and the characteristics and 
number of participants. You then compare this information to 
your initial objectives and determine whether there is a difference 
between objectives and actual implementation. This process will 
answer the question: Were program implementation objectives attained ? 

If there are differences between your objectives and your actual 
implementation, you can analyze your evaluation information to 
identify the reasons for the differences. This step answers the 
question: If not, why not? 
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You also can use your evaluation information to identify barriers 
encountered to implementation and factors that facilitated 
implementation. This information can be used to "tell the story" 
of your program's implementation. An example of how this 
information might be organized for a drug abuse prevention 
program for mnaway and homeless youth is provided in a table at 
the end of this chapter. The table represents an analysis of the 
program's measurable implementation objective concerning what 
the program plans to do. 

You may remember that the measurable objectives introduced as 
examples in this manual for what you plan to do for the drug 
abuse prevention program were the following: 

The program will provide eight drug abuse education class sessions per year. 

Each session will last for 2 weeks. 

Each 2-week session will involve 2 hours of classes per day. 

Classes will be held for 5 days of each week of the session. 

In the table, these measurable objectives appear in the first 
column. The actual program implementation information is 
provided in the second column. For this program, there were 
differences between objectives and actual implementation for three 
of the four measurable objectives. Column 3 notes the presence or 
absence of differences, and column 4 provides the reasons for 
those changes. 

Columns 5 and 6 in the table identify the barriers encountered and 
the facilitating factors. These are important to identify whether or 
not implementation objectives were attained. They provide the 
context for understanding the program and will help you interpret 
the results of your analyses. 

By reviewing the information in this table, you would be able to 
say the following things about your program: 

The program implemented only six drug abuse prevention 
sessions instead of the intended eight sessions. 

» The fewer than expected sessions were caused by a delay in 
startup time. 
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NOTE: It is important to 
understand that problems in 
attaining a particular program 
implementation objective are not 
a sign of program failure. Often, 
initial program implementation 
objectives are changed during the 
course of implementation in 
order to improve the program. 
The evaluation’s focus on 
attaining implementation 
objectives is only a framework for 
understanding the events that 
occurred during implementation 
of your program. 


» The delay was caused by the difficulty of recruiting and hiring 
qualified staff, which took longer than expected. 

» With staff now on board, we expect to be able to implement the 
full eight sessions in the second year. 

» Once staff were hired, the sessions were implemented smoothly 
because there were a number of volunteers who provided 
assistance in organizing special events and transporting participants 
to the events. 

Although the first two sessions were conducted for 2 weeks 
each, as intended, the remaining sessions were conducted for 
only 1 week. 

» The decreased duration of the sessions was caused by the 
difficulty of maintaining the youth's interest during the 2-week 
period. 

» Attendance dropped considerably during the second week, 
usually because of lack of interest, but sometimes because youth 
were moved to other placements or returned home. 

» Attendance during the first week was maintained because of the 
availability of youth residing in the shelter. 

For the first two sessions the class time was 2 hours per day, 
as originally intended. After the number of sessions was 
decreased, the class time was increased to 3 hours per day. 

» The increase was caused by the need to cover the curriculum 
material during the session. 

» The extensive experience of the staff, and the assistance of 
volunteers, facilitated covering the material during the 1-week 
period. 

» The youth's interest was high during the 1-week period. 

The classes were provided for 5 days during the 1-week 
period, as intended. 

» This schedule was facilitated by staff availability and the access to 
youth residing in the shelter. 
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» It was more difficult to get youth from crisis intervention services 
to attend for all 5 days. 

Information on this implementation objective will be expanded as 
you conduct a similar analysis of information relevant to the other 
implementation objectives of staffing (who will do it) and the 
population (number and characteristics of participants). 

As you can see, if this information is provided on an on-going 
basis, it will provide opportunities for the program to improve its 
implementation and better meet the needs of program participants. 

ANALYZING INFORMATION ABOUT PARTICIPANT 
OUTCOME OBJECTIVES 

The analysis of participant outcome information must be designed 
to answer two questions: 

Did the expected changes in participants' knowledge, attitudes, behavior, or 
awareness occur ? 

If changes occurred, were they the result of jour program’s interventions ? 

Another question that can be included in your analysis of 
participant outcome information is: 

Did some participants change more than others and, if so, what explains this 
difference? (For example, characteristics of the participants, types of 
interventions, duration of interventions, intensity of interventions, 
or characteristics of staff.) 

Your evaluation plan must include a detailed description of how 
you will analyze information to answer these questions. It is very 
important to know exactly what you want to do before you begin 
collecting data, particularly the types of statistical procedures 
that you will use to analyze participant outcome information. 

Understanding statistical procedures. Statistical procedures are 
used to understand changes occurring among participants as a 
group. In many instances, your program participants may vary 
considerably with respect to change. Some participants may change 
a great deal, others may change only slightly, and still others may 
not change or may change in an unexpected direction. Statistical 
procedures will help you assess the overall effectiveness of your 
program and its effectiveness with various types of participants. 
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Statistical procedures also are important tools for an evaluation 
because they can determine whether the changes demonstrated by 
your participants are the result of a chance occurrence or are 
caused by the variables (program or procedure) being assessed. 
This is called statistical significance. Usually, a change may be 
considered statistically significant (not just a chance occurrence) if 
the probability of its happening by chance is less than 5 in 100 
cases. However, in some situations, evaluators may set other 
standards for establishing significance, depending on the nature of 
the program, what is being measured, and the number of 
participants. Statistical significance is influenced by both the size of 
the effect and the size of the sample (number of participants). 

Another use for statistical procedures is determining the similarity 
between your treatment and nontreatment group members. This is 
particularly important if you are using a comparison group rather 
than a control group as your nontreatment group. If a comparison 
group is to be used to establish that participant changes were the 
result of your program's interventions and not some other factors, 
you must demonstrate that the members of the comparison group 
are similar to your participants in every key way except for 
program participation. 

Statistical procedures can be used to determine the extent of 
similarity of group members with respect to age, gender, 
socioeconomic status, marital status, race or ethnicity, or other 
factors. 

Statistical tests are a type of statistical procedure that examine the 
relationships among variables in an analysis. Some statistical tests 
include a dependent variable, one or more independent variables, 
and potential moderating or conditioning variables. 

Dependent variables are your measures of the knowledge, 
attitude, or behavior that you expect will change as a result of your 
program. For example, if you expect parents to increase their 
scores on an instmment measuring understanding of child 
development or effective parenting, the scores on that instrument 
are the dependent variable for the statistical analyses. 

Independent variables refer to your program interventions or 
elements. For example, the time of data collection (before and 
after program participation), the level of services or training, or the 
duration of services may be your independent variables. 
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NOTE: The types of statistical 
analyses that can be used for 
evaluation purposes are extensive 
and each test requires that the 
data being analyzed meet specific 
criteria. Also, there are different 
tests for different types of data. 
Unless you have an experienced 
evaluator as a member of your 
evaluation team, you will need 
assistance from an outside 
consultant in deciding the 
appropriate statistical analyses, 
setting up the data base, and 
conducting the statistical tests. 


Moderating or conditioning variables are those that may affect 
the relationship between the independent variable and the 
dependent variable. These are factors such as the participant's 
gender, socioeconomic status, age, race, or ethnicity. 

Most statistical tests assess the relationships among independent 
variables, dependent variables, and moderating variables. The 
specific question answered by most statistical tests is: Does the 
dependent variable vary as a function of levels of the independent 
variable? For example, do scores on an instrument measuring 
understanding of child development vary as a function of when the 
instrument was administered (before and after the program)? In 
other words, did attendance at your program's child development 
class increase parents' knowledge? 

Most statistical tests can also answer whether any other factors 
affected the relationship between the independent and dependent 
variables. For example, was the variation in scores from before to 
after the program affected by the ages of the persons taking the 
test, their socioeconomic status, their ethnicity, or other factors? 
The more independent and moderating variables you include in 
your statistical analyses, the more you will understand about your 
program's effectiveness. 

As an example, you could assess whether parents' scores on an 
instrument measuring understanding of child development differed 
as a result of the time of instrument administration (at intake and 
at program exit), the age of the parent, and whether or not they 
completed the full program. 

Suppose your statistical test indicates that, for your population as a 
whole, understanding of child development did not change 
significantly as a result of the time of instmment administration. 
That is, "program exit" scores were not significantly higher than 
"program intake" scores. This finding would presumably indicate 
that you were not successful in attaining this expected participant 
outcome. 

However, lack of a significant change among your 
participants as a group does not necessarily rule out program 
effectiveness. If you include the potential moderating variable of 
age in your analysis, you may find that older mothers (ages 25 to 
35) did demonstrate significant differences in before-and-after 
program scores but younger mothers (ages 17 to 24 years) did not. 
This would indicate that your program's interventions are 
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just as it is important to analyze 
the progress of participants who 
completed your program, it is 
also important to analyze the 
progress made by those who 
dropped out or did not complete 
the program entirely. This is 
significant information for your 
evaluation report. 


effective for the older mothers in your target population, but not 
for the younger ones. You may then want to implement different 
types of interventions for the younger mothers, or you may want 
to limit your program recmitment to older mothers, who seem to 
benefit from what you are doing. And you would not have known this 
without the evaluation ! 

If you added the variable of whether or not participants completed 
the full program, you may find that those who completed the 
program were more likely to demonstrate increases in scores than 
mothers who did not complete the program and, further, that 
older mothers were more likely to complete the program than 
younger mothers. Based on this finding, you may want to find out 
why the younger mothers were not completing the program so that 
you can develop strategies for keeping younger mothers in the 
program. 

USING THE RESULTS OF YOUR ANALYSES 

The results of your analyses can answer your initial evaluation 
questions. 

Are participant outcome objectives being attained ? 

If not, why not? 

What factors contributed to attainment of objectives? 

What factors ivere barriers to attainment of objectives? 

These questions can be answered by interpreting the results of the 
statistical procedures performed on the participant outcome 
information. However, to fully address these questions, you will 
also need to look to the results of the analysis of program 
implementation information. This will provide a context for 
interpreting statistical results. 

For example, if you find that one or more of your participant 
outcome objectives is not being attained, you may want to explain 
this finding. Sometimes you can look to your analysis of program 
implementation information to understand why this may have 
happened. You may find, for example, that your program was 
successful in attaining the outcome of an increase in parents' 
knowledge about child development, but was not successful in 
attaining the behavioral outcome of improved parenting skills. 
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In reviewing your program implementation information, you may 
find that some components of your program were successfully 
implemented as intended, but that the home-based counseling 
component of the program was not fully implemented as intended 
- and that the problems encountered in implementing the home- 
based counseling component included difficulty in recruiting 
qualified staff, extensive staff turnover in the counselor positions, 
and insufficient supervision for staff. Because the participant 
outcome most closely associated with this component was 
improving parenting skills, the absence of changes in this behavior 
may be attributable to the problems encountered in implementing 
this objective. 

The results of integrating information from your participant 
outcome and program implementation analyses are the content for 
your evaluation report. Ideally, evaluation reports should be 
prepared on an ongoing basis so that you can receive feedback on 
the progress of your evaluation and your program. The specified 
times for each report would depend on your need for evaluation 
information, the time frame for the evaluation, and the duration of 
the program. Chapter 9 provides more information on preparing 
an evaluation report. 
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SAMPLE TABLE FOR ANALYZING INFORMATION ON IMPLEMENTATION OBJECTIVES 


Implementation 

Objective 

Actual Implementation 

Differences? 

(Yes/No) 

If Yes, Reasons for 
Change 

Barriers Encountered 

Facilitating Factors 

8 drug abuse education 
class sessions per year 

6 drug abuse education 
class sessions the first 

Yes 

Delay in start up time 
during the first year 

Difficulty finding 
qualified staff 

Delay in curriculum 
development 

Agency experience in 
implementing similar 
types of programs 
Assistance of volunteers 
with the sessions 

Each session will last for 

2 weeks 

First 2 sessions lasted 
for 2 weeks, last 4 
sessions for 1 week 

Yes 

Participants could not 
consistently attend for 2 
weeks 

Youth lost interest 
during the second week 

Available participants in 
the shelter 

Each class will be for 2 
hours 

First 2 sessions, classes 
were for 2 hours each 
day; last 4 sessions 
were for 3 hours each 
day 

Yes 

Since the time was 
shortened, had to 
extend the intensity of 
classes to cover the 
curriculum material 

None 

Experienced staff able to 
cover the curriculum 
during the shortened 
time span 

Classes will be given for 

5 days of each week 

5 days a week 

No 


Problems with crisis 
intervention youth 
attending for all 5 days 

Staff availability 




IN THIS CHAPTER... 
Preparing reports for 
various audiences 
Tools for disseminating 
results 

Sample outline 



How Can You 
Report What You 
Have Learned? 

A n evaluation report is an important document. It 
integrates what you have learned about your program 
from the evaluation. However, it is vital to understand 
that there are different ways of reporting evaluation 
information, depending on how you want to use the report and 
who your audience will be. In this chapter, we suggest preparing 
evaluation reports that are appropriate for a range of uses. A 
program evaluation report can do the following: 

■ Guide management decisions by identifying areas in which 
changes may be needed for future implementation Tell the 
"story" of program implementation and demonstrate the 
impact of the program on participants 

■ Advocate for your program with potential funders or with 
other community agencies to encourage referrals 

■ Help advance the field of human services 

These uses suggest that various audiences for an evaluation report 
might include program staff and agency directors, program 
funders, potential funders, agency boards, other community 
agencies, and local and national organizations that advocate for 
individuals like your program participants or for programs such as 
yours. 

Whatever type of report you plan to develop, remember that it is 
critical to report negative results, as well as significant ones. There 
is as much to learn from program approaches or models that do 
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You may also want to 
publish the results of 
your evaluation in a 
professional journal. 
However, this chapter 
will not address this 
type of evaluation 
report. The member of 
your evaluation team 
who is an experienced 
evaluator should be 
able to help you 
prepare a report for 
publication. 


not work as there is from those that work. Negative results should 
not be thought of as shameful. Efforts to change knowledge, 
attitudes, and behaviors through programmatic interventions are 
not always going to work. It is also important to present results 
that may not be conclusive, but show promise and warrant 
additional study. For example, if mothers over the age of 25 
seemed to improve their parenting skills after receiving home- 
based support services, this is worth presenting so future 
evaluation can explore this further. Currently, so little is known 
about what does and does not work that any information on these 
issues greatly increases knowledge in the field. 

PREPARING AN EVALUATION REPORT FOR 
PROGRAM FUNDERS 

The report to program funders will probably be the most 
comprehensive report you prepare. Often program funders will use 
your report to demonstrate the effectiveness of their grant 
initiatives and to support allocation of additional moneys for 
similar programs. A report that is useful for this purpose will need 
to include detailed information about the program, the evaluation 
design and methods, and the types of data analyses conducted. 

A sample outline for an evaluation report for program funders is 
provided in this chapter. The outline is developed for a "final 
report" and assumes all the information collected on your program 
has been analyzed. However, this outline may also be used for 
interim reports, with different sections completed at various times 
during the evaluation and feedback provided to program personnel 
on the ongoing status of the evaluation. 

PREPARING AN EVALUATION REPORT FOR 
PROGRAM STAFF AND AGENCY PERSONNEL 

An evaluation report for program staff and agency personnel may be 
used to support management decisions about ongoing or future 
program efforts. This type of report may not need to include as much 
detail on the evaluation methodology but might focus instead on 
findings. The report could include the information noted in outline 
Sections II E (description of results of analysis of implementation 
information), III D (discussion of issues that affected the outcome 
evaluation and how they were addressed), III F (results of data analysis 
on participant outcome information), III G (discussion of results), and 
IV C (discussion of potential relationships between implementation 
and outcome evaluation results). 
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EXECUTIVE 
SUMMARY. Your 
report should be 
accompanied by an 
executive summary that 
summarizes the key 
evaluation methods and 
results, so that readers 
will not have to review 
all of the details of the 
report if they do not 
have the time. 


PREPARING AN EVALUATION REPORT FOR 
POTENTIAL FUNDERS AND ADVOCACY 
ORGANIZATIONS 

It is unlikely that potential funders (including State legislatures and 
national and local foundations) or advocacy organizations will want to 
read a lengthy report. In a report for this audience, you may want to 
focus on the information provided in Section IV of the outline. This 
report would consist of only a summary of both program 
implementation and participant outcome objectives and a discussion 
of the relationships between implementation policies, practices, 
procedures, and participant outcomes. 

DISSEMINATING THE RESULTS OF YOUR 
EVALUATION 

In addition to producing formal evaluation reports, you may want to 
take advantage of other opportunities to share what you have learned 
with others in your community or with the field in general. You might 
want to consider drafting letters to community health and social 
services agencies or other organizations that may be interested in the 
activities and results of your work. Other ways to let people know 
what you have done include the following: 

■ Producing press releases and articles for local professional 
publications, such as newsletters and journals 

■ Making presentations at meetings on the results of your program 
at the local health department, university or public library, or other 
setting 

■ Listing your evaluation report or other evaluation-related 
publications in relevant databases, on electronic bulletin boards, 
and with clearinghouses 

■ Making telephone calls and scheduling meetings with similar 
programs to share your experience and results 

Many of the resource materials listed in the appendix of this manual 
contain ideas and guidelines for producing different types of 
informational materials related to evaluations. 
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Sample Outline 
Final Evaluation Report 

EXECUTIVE SUMMARY 


I. Introduction: General Description of the Project (1 page) 

A. Description of program components, including services or 
training delivered and target population for each service 

B. Description of collaborative efforts (if relevant), including 
the agencies participating in the collaboration and their 
various roles and responsibilities in the project 

C. Description of strategies for recruiting program participants 
(if relevant) 

D. Description of special issues relevant to serving the 
project's target population (or providing education and 
training to participants) and plans to address them 

1. Agency and staffing issues 

2. Participants' cultural background, socioeconomic 
status, literacy levels, and other characteristics 

II. Evaluation of Program Implementation Objectives 

A. Description of the project's implementation 
objectives(measurable objectives) > 

1. What you planned to do (planned 
services/interventions/training/education; 
duration and intensity of each 
service/intervention/training period) 

2. Whom you planned to have do it (planned staffing 
arrangements and qualifications/characteristics of 
staff) 

3. Target population (intended characteristics and 
number of members of the target population to be 
reached by each service/intervention/training/ 
education effort and how you planned to recruit 
participants) 

4. Description of the project's objectives for 
collaborating with community agencies 

a. Planned collaborative arrangements 

b. Services/interventions/training provided 
by collaborating agencies 

B. Statement of evaluation questions (Were program 
implementation objectives attained? If not, why not? What 
were the barriers to and facilitators of attaining 
implementation objectives?) 



HOW CAN YOU REPORT WHAT YOU HAVE LEARNED? 


c. 


Examples: 

How successful was the project in implementing a 
parenting education class for mothers with substance 
abuse problems? What were the policies, practices, and 
procedures used to attain this objective? What were the 
barriers to, and facilitators of attaining this objective? 

How successful was the project in recruiting the intended 
target population and serving the expected number of 
participants? What were the policies, practices, and 
procedures used to recruit and maintain participants in the 
project? What were the barriers to, and facilitators of 
attaining this objective? 

How successful was the project in developing and 
implementing a multidisciplinary training curriculum? What 
were the practices and procedures used to develop and 
implement the curriculum? What were the barriers to, and 
facilitators of attaining this objective? 

How successful was the project in establishing 
collaborative relationships with other agencies in the 
community? What were the policies, practices, and 
procedures used to attain this objective? What were the 
barriers to, and facilitators of attaining this objective? 

D. Description of data collection methods and data collected 
for each evaluation question 

1. Description of data collected 

2. Description of methodology of data collection 

3. Description of data sources (such as project 
documents, project staff, project participants, and 
collaborating agency staff) 

4. Description of sampling procedures 

E. Description of data analysis procedures 

F. Description of results of analysis 

1. Statement of findings with respect to each 
evaluation question 

Examples: 

The project's success in attaining the objective 

The effectiveness of particular policies, practices, 
and procedures in attaining the objective 

The barriers to and facilitators of attainment of 
the objective 
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2. Statement of issues that may have affected the 
evaluation's findings 

Examples: 

The need to make changes in the evaluation 
because of changes in program implementation or 
characteristics of the population served 

Staff turnover in the project resulting in 
inconsistent data collection procedures 

Changes in evaluation staff 

III. Evaluation of Participant Outcome Objectives 

A. Description of participant outcome objectives (in 
measurable terms) 

1. What changes were participants expected to 
exhibit as a result of their participation in each 
service/intervention/training module provided by 
the project? 

2. What changes were participants expected to 
exhibit as a result of participation in the project in 
general? 

3. What changes were expected to occur in the 
community's service delivery system as a result of 
the project? 

B. Statement of evaluation questions, evaluation design, and 
method for assessing change for each question 

Examples: 

How effective was the project in attaining its expected 
outcome of decreasing parental substance abuse? How 
was this measured? What design was used to establish 
that a change occurred and to relate the change to the 
project's interventions (such as pre intervention and 
post intervention, control groups, comparison groups, 
etc.)? Why was this design selected? 

How effective was the project in attaining its expected 
outcome of increasing children's self-esteem? How was 
this measured? What design was used to establish that 
a change occurred and to relate the change to the 
project's interventions? Why was this design selected? 

How effective was the project in increasing the 
knowledge and skills of training participants? How was 
this measured? What design was used to establish that 
a change occurred and to relate the change to the 
project's interventions? Why was this design selected? 

C. Discussion of data collection methods (for each evaluation 
question) 


1. Data collected 
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2. Method of data collection 
Examples: 

Case record reviews 
Interviews 

Self-report questionnaires or inventories (if you 
developed an instrument for this evaluation, 
attach a copy to the final report) 

Observations 

3. Data sources (for each evaluation question) and 
sampling plans, when relevant 

D. Discussion of issues that affected the outcome evaluation 
and how they were addressed 

1. Program-related issues 

a. Staff turnover 

b. Changes in target population 
characteristics 

c. Changes in services/interventions during 
the course of the project 

d. Changes in staffing plans 

e. Changes in collaborative arrangements 

f. Characteristics of participants 

2. Evaluation-related issues 

a. Problems encountered in obtaining 
participant consent 

b. Change in numbers of participants served 
requiring change in analysis plans 

c. Questionable cultural relevance of 
evaluation data collection instruments 
and/or procedures 

d. Problems encountered due to participant 
attrition 

E. Procedures for data analyses 

F. Results of data analyses 

1. Significant and negative analyses results 
(including statement of established level of 
significance) for each outcome evaluation 
question 

2. Promising, but inconclusive analyses results 
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3. Issues/problems relevant to the analyses 
Examples: 

Issues relevant to data collection procedures, 
particularly consistency in methods and 
consistency across data collectors 

Issues relevant to the number of participants 
served by the project and those included in the 
analysis 

Missing data or differences in size of sample for 
various analyses 

G. Discussion of results 

1. Interpretation of results for each evaluation 
question, including any explanatory information 
from the process evaluation 

a. The effectiveness of the project in 
attaining a specific outcome objective 

b. Variables associated with attainment of 
specific outcomes, such as characteristics 
of the population, characteristics of the 
service provider or trainer, duration, or 
intensity of services or training, and 
characteristics of the service or training 

2. Issues relevant to interpretation of results 
IV. Integration of Process and Outcome Evaluation Information 

A. Summary of process evaluation results 

B. Summary of outcome evaluation results 

C. Discussion of potential relationships between program 
implementation and participant outcome evaluation results 

Examples: 

Did particular policies, practices, or procedures used to 
attain program implementation objectives have different 
effects on participant outcomes? 

How did practices and procedures used to recruit and 
maintain participants in services affect participant 
outcomes? 

What collaboration practices and procedures were found to 
be related to attainment of expected community 
outcomes? 

Were particular training modules more effective than 
others in attaining expected outcomes for participants? If 
so, what were the features of these modules that may 
have contributed to their effectiveness (such as 
characteristics of the trainers, characteristics of the 
curriculum, the duration and intensity of the services)? 
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V. Recommendations to Program Administrators or Funders for Future 
Program and Evaluation Efforts 

Examples: 

Based on the evaluation findings, it is recommended that the 
particular service approach developed for this program be used 
to target mothers who are 25 years of age or older. Younger 
mothers do not appear to benefit from this type of approach. 

The evaluation findings suggest that traditional educational 
services are not as effective as self-esteem building services in 
promoting attitude changes among adolescents regarding 
substance abuse. 1/1/e recommend that future program 
development focus on providing these types of services to 
youth at risk for substance abuse. 

Based on the evaluation findings, it is recommended that 
funders provide sufficient funding for evaluation that will permit 
a long-term follow-up assessment of participants. The kinds of 
participant changes that the program may bring about may not 
be observable until 3 or 6 months after they leave the 
program. 
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Glossary 


baseline data - Initial information on program participants or other program aspects collected prior to 
receipt of services or program intervention. Baseline data are often gathered through intake interviews and 
observations and are used later for comparing measures that determine changes in your participants, program, 
or environment. 

bias - (refers to statistical bias). Inaccurate representation that produces systematic error in a research finding. 
Bias may result in overestimating or underestimating certain characteristics of the population. It may result from 
incomplete information or invalid collection methods and may be intentional or unintentional. 

comparison group - Individuals whose characteristics (such as race/ethnicity, gender, and age) are 
similar to those of your program participants. These individuals may not receive any services, or they may 
receive a different set of services, activities, or products. In no instance do they receive the same service(s) as 
those you are evaluating. As part of the evaluation process, the experimental (or treatment) group and the 
comparison group are assessed to determine which type of services, activities, or products provided by your 
program produced the expected changes. 

confidentiality - Since an evaluation may entail exchanging or gathering privileged or sensitive 
information about individuals, a written form that assures evaluation participants that information provided will 
not be openly disclosed nor associated with them by name is important. Such a form ensures that their privacy 
will be maintained. 

consultant - An individual who provides expert or professional advice or services, often in a paid capacity. 

control group - A group of individuals whose characteristics (such as race/ ethnicity, gender, and age) are 
similar to those of your program participants, but do not receive the program (services, products, or activities) 
you are evaluating. Participants are randomly assigned to either the treatment (or program) group and the 
control group. A control group is used to assess the effect of your program on participants as compared to 
similar individuals not receiving the services, products, or activities you are evaluating. The same information is 
collected for people in the control group as in the experimental group. 

cost-benefit analysis - A type of analysis that involves comparing the relative costs of operating a 
program (program expenses, staff salaries, etc.) to the benefits (gains to individuals or society) it generates. For 
example, a program to reduce cigarette smoking would focus on the difference between the dollars expended 
for converting smokers into nonsmokers with the dollar savings from reduced medical care for smoking related 
disease, days lost from work, and the like. 

cost effectiveness analysis - A type of analysis that involves comparing the relative costs of 
operating a program with the extent to which the program met it goals and objectives. For example, a program 
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to reduce cigarette smoking would estimate the dollars that had to be expended in order to convert each 
smoker into a nonsmoker. 

cultural relevance - Demonstration that evaluation methods, procedures, and or instruments are 
appropriate for the culture(s) to which they are applied. (Other terms include cultural competency, cultural 
sensitivity). 

culture - The shared values, traditions, norms, customs, arts, history, institutions, and experience of a group 
of people. The group may be identified by race, age, ethnicity, language, national origin, religion, or other social 
category or grouping. 

data - Specific information or facts that are collected. A data item is usually a discrete or single measure. 
Examples of data items might include age, date of entry into program, or reading level. Sources of data may 
include case records, attendance records, referrals, assessments, interviews, and the like. 

data analysis - The process of systematically applying statistical and logical techniques to describe, 
summarize, and compare data collected. 

data collection instruments - Forms used to collect information for your evaluation. Forms may 
include interview instruments, intake forms, case logs, and attendance records. They may be developed 
specifically for your evaluation or modified from existing instruments. A professional evaluator can help select 
those that are most appropriate for your program. 

data collection plan - A written document describing the specific procedures to be used to gather the 
evaluation information or data. The plan describes who collects the information, when and where it is collected, 
and how it is to be obtained. 

database - An accumulation of information that has been systematically organized for easy access and 
analysis. Databases typically are computerized. 

design - The overall plan and specification of the approach expected in a particular evaluation. The design 
describes how you plan to measure program components and how you plan to use the resulting measurements. 
A pre- and post-intervention design with or without a comparison or control group is the design needed to 
evaluate participant outcome objectives. 

evaluation - A systematic method for collecting, analyzing, and using information to answer basic 
questions about your program. It helps to identify effective and ineffective services, practices, and approaches. 

evaluator - An individual trained and experienced in designing and conducting an evaluation that uses 
tested and accepted research methodologies. 

evaluation plan - A written document describing the overall approach or design you anticipate using to 
guide your evaluation. It includes what you plan to do, how you plan to do it, who will do it, when it will be 
done, and why the evaluation is being conducted. The evaluation plan serves as a guide for the evaluation. 
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evaluation team -The individuals, such as the outside evaluator, evaluation consultant, program 
manager, and program staff who participate in planning and conducting the evaluation. Team members assist in 
developing the evaluation design, developing data collection instruments, collecting data, analyzing data, and 
writing the report. 

exit data - Information gathered after an individual leaves your program. Exit data are often compared to 
baseline data. For example, a Head Start program may complete a developmental assessment of children at the 
end of the program year to measure a child's developmental progress by comparing developmental status at the 
beginning and end of the program year. 

experimental group - A group of individuals receiving the treatment or intervention being evaluated or 
studied. Experimental groups (also known as treatment groups) are usually compared to a control or 
comparison group. 

focus group - A group of 7-10 people convened for the purpose of obtaining perceptions or opinions, 
suggesting ideas, or recommending actions. A focus group is a method of collecting data for evaluation 
purposes. 

formative evaluation - A type of process evaluation of new programs or services that focuses on 
collecting data on program operations so that needed changes or modifications can be made to the program in 
its early stages. Formative evaluations are used to provide feedback to staff about the program components that 
are working and those that need to be changed. 

immediate outcomes - The changes in program participants, knowledge, attitudes, and behavior that 
occur early in the course of the program. They may occur at certain program points, or at program completion. 
For example, acknowledging substance abuse problems is an immediate outcome. 

impact evaluation - A type of outcome evaluation that focuses on the broad, longer-term impacts or 
results of a program. For example, an impact evaluation could show that a decrease in a community's overall 
infant mortality rate was the direct result of a program designed to provide early prenatal care. 

in-kind service Time or services donated to your program. 

informed consent - A written agreement by program participants to voluntarily participate in an 
evaluation or study after having been advised of the purpose of the study, the type of information being 
collected, and how the information will be used. 

instrument - A tool used to collect and organize information. Includes written instruments or measures, 
such as questionnaires, scales, and tests. 

intermediate outcomes - Results or outcomes of a program or treatment that may require some time 
before they are realized. For example, part-time employment would be an intermediate outcome of a program 
designed to assist at-risk youth in becoming self-sufficient. 
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internal resources - An agency's or organization's resources including staff skills and experiences and 
any information you already have available through current program activities. 

intervention - The specific services, activities, or products developed and implemented to change or 
improve program participants' knowledge, attitudes, behaviors, or awareness. 

logic model - A diagram showing the logic or rationale underlying your particular program. In other 
words, it is a picture of a program that shows what it is supposed to accomplish. A logic model describes the 
links between program objectives, program activities, and expected program outcomes. 

management information system (MIS) - An information collection and analysis system, 
usually computerized, that facilitates access to program and participant information. It is usually designed and 
used for administrative purposes. The types of information typically included in an MIS are service delivery 
measures, such as session, contacts, or referrals; staff caseloads; client sociodemographic information; client 
status; and treatment outcomes. Many MIS can be adapted to meet evaluation requirements. 

measurable terms - Specifying, through clear language, what it is you plan to do and how you plan to 
do it. Stating time periods for activities, "dosage" or frequency information (such as three 1-hour training 
sessions), and number of participants helps to make project activities measurable. 

methodology - The way in which you find out information; a methodology describes how something will 
be (or was) done. The methodology includes the methods, procedures, and techniques used to collect and 
analyze information. 

monitoring - The process of reviewing a program or activity to determine whether set standards or 
requirements are being met. Unlike evaluation, monitoring compares a program to an ideal or exact state. 

objective - A specific statement that explains how a program goal will be accomplished. For example, an 
objective of the goal to improve adult literacy could be to provide tutoring to participants on a weekly basis for 
6 months. An objective is stated so that changes, in this case, an increase in a specific type of knowledge, can be 
measured and analyzed. Objectives are written using measurable terms and are time-limited. 

Outcome - Outcomes are a result of the program, services, or products you provide and refer to changes in 
knowledge, attitude, or behavior in participants. They are referred to as participant outcomes in this manual. 

outcome evaluation - Evaluation designed to assess the extent to which a program or intervention 
affects participants according to specific variables or data elements. These results are expected to be caused by 
program activities and tested by comparison of results across sample groups in the target population. Also 
known as impact and summative evaluation. 

outcome objectives - The changes in knowledge, attitudes, awareness, or behavior that you expect to 
occur as a result of implementing your program component, service, or activity. Also known as participant 
outcome objectives. 
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outside evaluator - An evaluator not affiliated with your agency prior to the program evaluation. Also 
known as a third-party evaluator. 

participant - An individual, family, agency, neighborhood, community, or State, receiving or participating 
in services provided by your program. Also known as a client or target population group. 

pilot test - Preliminary test or study of your program or evaluation activities to try out procedures and make 
any needed changes or adjustments. For example, an agency may pilot test new data collection instruments that 
were developed for the evaluation. 

posttest - A test or measurement taken after a service or intervention takes place. It is compared with the 
results of a pretest to show evidence of the effects or changes as a result of the service or intervention being 
evaluated. 

pretest - A test or measurement taken before a service or intervention begins. It is compared with the 
results of a posttest to show evidence of the effects of the service or intervention being evaluated. A pretest can 
be used to obtain baseline data. 

process evaluation - An evaluation that examines the extent to which a program is operating as 
intended by assessing ongoing program operations and whether the targeted population is being served. A 
process evaluation involves collecting data that describes program operations in detail, including the types and 
levels of services provided, the location of service delivery, staffing; sociodemographic characteristics of 
participants; the community in which services are provided, and the linkages with collaborating agencies. A 
process evaluation helps program staff identify needed interventions and/ or change program components to 
improve service delivery. It is also called formative or implementation evaluation. 

program implementation objectives - What you plan to do in your program, component, or 
service. For example, providing therapeutic child care for 15 children, giving them 2 hot meals per day, are 
referred to as program implementation objectives. 

qualitative data - Information that is difficult to measure, count, or express in numerical terms. For 
example, a participant's impression about the fairness of a program rule/ requirement is qualitative data. 

quantitative data - Information that can be expressed in numerical terms, counted or compared on a 
scale. For example, improvement in a child's reading level as measured by a reading test. 

random assignment - The assignment of individuals in the pool of all potential participants to either 
the experimental (treatment) or control group in such a manner that their assignment to a group is determined 
entirely by chance. 

reliability - Extent to which a measurement (such as an instrument or a data collection procedure) 
produces consistent results over repeated observations or administrations of the instmment under the same 
conditions each time. It is also important that reliability be maintained across data collectors; this is call 
interrater reliability. 


100 



GLOSSARY 


sample - A subset of participants selected from the total study population. Samples can be random (selected 
by chance, such as every 6th individual on a waiting list) or nonrandom (selected purposefully, such as all 2-year 
olds in a Head Start program). 

Standardized instruments - Assessments, inventories, questionnaires, or interviews, that have been 
tested with a large number of individuals and are designed to be administered to program participants in 
consistent manner. Results of tests with program participants can be compared to reported results of the tests 
used with other populations. 

statistical procedures - The set of standards and rules based in statistical theory, by which one can 
describe and evaluate what has occurred. 

statistical test Type of statistical procedure, such as a t-test or Z-score, that is applied to data to 
determine whether your results are statistically significant (i.e., the outcome is not likely to have resulted by 
chance alone). 

SUmmative evaluation - A type of outcome evaluation that assesses the results or outcomes of a 
program. This type of evaluation is concerned with a program's overall effectiveness. 

treatment group - Also called an experimental group, a treatment group is composed of a group of 
individuals receiving the services, products, or activities (interventions) that you are evaluating. 

validity - The extent to which a measurement instrument or test accurately measures what it is supposed to 
measure. For example, a reading test is a valid measure of reading skills, but is not a valid measure of total 
language competency. 

variables Specific characteristics or attributes, such as behaviors, age, or test scores, that are expected to 
change or vary. For example, the level of adolescent dmg use after being exposed to a drug prevention program 
is one variable that may be examined in an evaluation. 
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EVALUATION MANUALS AND GUIDES 

Centers for Disease Control and Prevention (1999). Framework for program evaluation in public health. 
MMWR, 48(RR1 1 ), 1-40. Retrieved from http:/ /www.cdc.gov/mmwr/preview/mmwrhtml/rr4811al.htm 

Thompson, N. J. & McClintock, H. O. (2000). Demonstrating Your Program's Worth: A Primer on Evaluation for 
Programs to Prevent Unintentional Injury. Atlanta: Centers for Disease Control and Prevention, National Center for 
Injury Prevention and Control. Retrieved from http:/ /www.cdc.gov/ ncipc / pub-res/ dypw/index.htm 

United Way Outcome Measurement Resource Network (1996). Measuring program outcomes: A practical approach. 

Alexandria, VA: United Way of America. Retrieved from 

http:/ /www.liveunited.org/ Outcomes/Resources/MPO/index.cfm 

U. S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the 
Director, Office of Strategy and Innovation (2005). Introduction to program evaluation for public health programs: A self- 
study guide. Atlanta, GA: Centers for Disease Control and Prevention. Retrieved from 
http:/ /www.cdc.gov/ eval/ evalguide.pdf 

W. K. Kellogg Foundation (1998). E valuation handbook. Battle Creek, MI: W. K. Kellogg Foundation. Retrieved 
from 

http:/ /www.wkkf.org/default.aspx?tabid=101&CID=281&CatID=281&ItemID=2810770&NID=20&Langu 
ageID=0 

W. K. Kellogg Foundation (2004). Logic model development guide. Battle Creek, MI: W. K. Kellogg Foundation. 
Retrieved from 

http:/ /www.wkkf.org/default.aspx?tabid=101&CID=281&CatID=281&ItemID=2813669&NID=20&Langu 
ageID=0 
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ONLINE EVALUATION TOOLKITS 

Bureau of Justice Assistance, Office of Justice Programs, U.S. Department of Justice 

Center for Program Evaluation and Performance Measurement 
http:/ /www.ojp.usdoj.gov/BJA/ evaluation/ 

Center for Research and Evaluation on Abstinence Education/U.S. Department of Health and Human 
Services 


Center for Research and Evaluation on Abstinence Education 

http:/ / abstinenceevaluation.hhs.gov/ tiki-index.php 

Child Welfare Information Gateway, a service of the Children's Bureau, Administration for Children and 
Families, U.S. Department of Health and Human Services 

Evaluation Toolkit and Logic Model Builder 

http:/ /www.childwelfare.gov/ preventing/ developing/ toolkit/ 

Steps in Evaluating Prevention Programs 

http:/ /www.childwelfare.gov/ preventing/ evaluating/ steps.cfm 

FRIENDS National Resource Center for Community-Based Child Abuse Prevention (CBCAP), a service of 
the Children's Bureau, Administration for Children and Families, U.S. Department of Health and Human 
Services 


Evaluation Toolkit 

http:/ /www.friendsnrc.org/ outcome/ toolkit/index.htm 

OTHER ONLINE RESOURCES 

American Evaluation Association 

■ Online Resources 

http:/ / www.eval.org/resources.asp 

Has listings of and links to: 

■ Professional groups of interest to evaluators; 

■ Sites containing compilations of tools and instruments of use to evaluators; 

■ Evaluation consultants; 

■ Email-based discussion lists (listservs) focusing on evaluation and/ or evaluation-related 
methodologies; 

■ Foundations funding applied research; 

■ Evaluation divisions/ units of government entities as well as evaluation units of non- 
government organizations (NGOs); 

■ Online handbooks and texts; 
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■ Products for developing and administering surveys as well as scanning survey/ form 
results; 

■ Products for analyzing qualitative data; 

■ University/ academic departments and centers that focus a significant portion of their 
work in the area of evaluation; and 

■ University and college programs that offer graduate-level programs and/ or certificates in 
evaluation. 

Centers for Disease Control and Prevention (CDC) Evaluation Working Group 

■ Resources 

http:/ /www.cdc.gov/ eval/ resources.htm 

Has links to resources related to: 

■ Evaluation ethics, principles, and standards; 

■ Organizations, societies, foundations, and associations; 

■ Journals and on-line publications; 

■ Step-by-step manuals; 

■ Logic model resources; 

■ Planning and performance improvement tools; and 

■ Reports and publications. 

National Network of Libraries of Medicine 

■ Outreach Evaluation Resource Center 
http:// nnlm.gov/ evaluation/ 

Resources include the guide Measuring the Difference: Guide to Planning and Devaluating Health Information 
Outreach, as well as supplemental booklets in the series Planning and Evaluating Health Information 
Outreach Projects. Also includes links to other tools and resources for evaluation and data collection. 

SRI International 

■ Online Evaluation Resource Library 
http:/ / oerl.sri.com/ 

Resources include collections of evaluation plans, instruments, and reports, which are organized into 
project categories, such as curriculum development and teacher education. Also includes professional 
development modules that can be used to better understand and utilize the materials made available. 
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CLEARINGHOUSES AND ORGANIZATIONS 


Healthflnder.gov 

http:/ / www.healthfinder.gov/ 

For healthfinder.gov’s list of Federal clearinghouses, click on “Find Services and Information,” 
then under “Find a Health Organization: Browse by Type of Organization,” click on “Federal 
Clearinghouses.” 

Within the “Find Services and Information” section of healthfinder.gov, you will also find lists of 
other types of organizations, including Federal agencies, health and human services 
clearinghouses, nonprofit organizations, professional organizations, and state health and human 
services. The organizations cover a broad range of health topics, including abstinence, adoption, 
child abuse and neglect, child care, child protection, child safety, child support, child welfare, early 
childhood education, family services, parenting, teenage pregnancy, and youth-at-risk, among 
many others. 

Selected Federal Clearinghouses 

Child Welfare Information Gateway 

Children's Bureau/ACYF 

1250 Maryland Avenue, SW 

Eighth Floor 

Washington, DC 20024 

ToU free: (800) 394-3366 

Phone: (703) 385-7565 

Fax: (703) 385.3206 

Email: info@cliildwelfare.gov 

http:/ /www.childwelfare.gov/ 

Early Childhood Learning and Knowledge Center (ECLKC) 

Administration for Children and Families 
Office of Head Start (OHS) 

8th Floor Portal Building 
Washington, DC 20024 
ToU free: (866) 763-6481 
Email: askus@headstartinfo.org 
http:/ / eclkc.ohs.acf.hhs.gov/hslc 

Education Resources Information Center (ERIC) 

ERIC Project 

c/ o Computer Sciences Corporation 
655 15th St. NW, Suite 500 
Washington, DC 20005 
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Phone: (800 ) 538-3742 
http:/ /www.eric.ed.gov/ 

Health Resources and Service Administration (HRSA) Information Center 

P.O. Box 2910 

Merrifield, VA 22118 

ToU free: (888) 275-4772 

TTY/TTD: (877) 489-4772 

Fax: (703) 821-2098 

http:/ / www.ask.hrsa.gov/MCH.cfm 

Email: ask@hrsa.gov 

Maternal and Child Health Information Resource Center (MCHIRC) 

1200 18th Street, NW, Suite 700 

Washington, DC 20036 

Phone: (202) 842-2000 

Fax: (202) 728-9469 

Email: mcliirc@altarum.org 

http:/ / www.mchb.hrsa.gov/mchirc/ 

National Child Care Information and Technical Assistance Center (NCCIC) 

10530 Rosehaven St., Suite 400 

Fairfax, VA 22030 

Phone: (800) 616-2242 

Fax: (800) 716-2242 

TTY: (800) 516-2242 

Email: info@nccic.org 

http:// nccic.acf.hhs.gov 

National Clearinghouse on Families and Youth (NCFY) 

P.O. Box 13505 

Silver Spring, Maryland 2091 1-3505 
Phone: (301) 608-8098 
Fax: (301) 608-8721 
http:/ /www.ncfy.com/ 

Substance Abuse and Mental Health Services Administration (SAMHSA) 

National Clearinghouse for Alcohol and Drug Information 

P.O. Box 2345 

Rockville, MD 20847-2345 

Phone: 1-877-SAMHSA-7 

Hablamos espanol: 1-877-767-8432 

TDD: 1-800-487-4889 

Fax: (240) 221-4292 

http:// ncadi.samhsa.gov/ about/ aboutncadi.aspx 
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LOCATING INFORMATION COLLECTION INSTRUMENTS/MEASURES 

■ Tests in Print (TIP) Series 

Publisher: Buros Institute of Mental Measurements 

TIP is a comprehensive bibliography of all known commercially available tests that are currently in 
print in the English language. Each TIP entry includes the test title, test purpose, test publisher, in- 
print status, price, test acronym, intended test population, administration times, publication date(s), 
and test author(s). Detailed information regarding the individual tests, such as psychometric 
information and reviews of the tests, are not included in the TIP entries. 

■ Tests: A Comprehensive Reference for Assessments in Psychology, Education, and Business 

Publisher: PRO-ED, Inc. 

Tests is a comprehensive bibliography of all tests available in the English language covering 
psychology, education, and business. Each entry includes: test title, authors’ names, copyright date, 
primary publisher, population for which the test is intended, a purpose statement, a brief description 
highlighting the test’s major features, format information, scoring method, and relevant cost and 
availability information. Detailed information regarding the individual tests, such as psychometric 
information and reviews of the tests, are not included in the entries. 

■ Mental Measurements Yearbook (MMY) Series 

Publisher: Buros Institute of Mental Measurements 

Each MMY test entry includes descriptive information (such as the test name, intended population, 
publication dates, forms and prices, test author, and publisher); additional information on the extent 
to which reliability, validity, norming data, scoring and reporting services, and foreign language 
versions are available; and one or two reviews of the test and testing materials. To be reviewed in the 
MMY a test must be commercially available, be published in the English language, and be new, 
revised, or widely used since it last appeared in the MMY series. Tests also must provide sufficient 
documentation supporting their technical quality to meet criteria for review. 

■ Test Critiques 

Publisher: PRO-ED, Inc. 

Test Critiques contains supplemental information regarding the tests that are cataloged in Tests, which is 
also published by PRO-ED. This information includes psychometric information such as reliability, 
validity, and norm development, as well as reviews from experts regarding the technical adequacy of 
the test. 
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Test Reviews Online 

http://buros.unl.edu/buros/jsp/ search.jsp 

Test Reviews Online provides test reviews exactly as they appear in the MMY series. In addition, 
monthly updates are provided from the latest test review database. For a small fee, users may 
download reviews for over 2,500 tests that include specifics on test purpose, population, publication 
date, administration time, and descriptive test critiques. 

TestTink, the Test Collection at educational Testing Service (ETS) 
http:/ /204.50.92.130/ETS_Test_Collection/Portal.aspx 

The Test Collection at ETS is a database of more than 25,000 tests and other measurement devices 
most of which were created by authors outside ETS. 

Ericae.net Clearinghouse on Assessment and Evaluation 

http:/ / ericae.net/ freetests.htm 

This database contains tests and instruments that have been appended to or contained in reports in 
the ERIC database or to journal articles. These tests can be readily used for research and for non- 
commercial purposes. 


Bridges, L. J., Berry, D., Calkins, J., Margie, N. G., Cochran, S., Ling, T., & Zaslow, M. (2003). Early 
childhood education and school readiness: Conceptual models, constructs, and measures: Profiles of early 
childhood measures. Washington, DC: Child Trends. 

FRIENDS National Resource Center for Community-Based Child Abuse Prevention (CBCAP). 

Outcome Accountability: Compendium of Annotated Measurement Tools 
http:/ / www.friendsnrc.org/ outcome/ toolkit/ annot.htm 

Halle, T., & Vick, J. E. (2007). Quality in Early Childhood Care and Education Settings: A Compendium 
of Measures. Washington, DC: Prepared by Child Trends for the Office of Planning, Research and 
Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. 

EVALUATION CONSULTANTS 


American Evaluation Association 
http:/ / www.eval.org/ 

Has a listing of association members who are available for evaluation consulting. 

American Sociological Association 
http:/ /www.asanet.org/ 
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Association for Public Policy Analysis and Management 
http:/ / www.appam.org/ 

Child Welfare League of America 
http:/ /www. cwla.org/ 

National Legislative Program Evaluation Society (NLPES) 
http:/ / www.ncsl.org/nlpes/ 
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